Looking at the data

We plot the data and can see that there is no obvious large difference between the debt versions.

d.both_completed %>%
  ggplot(aes(high_debt_version, fill = documentation)) +
  geom_bar(position = "fill") +
  scale_y_reverse() +
  scale_fill_manual("legend", values = c("Correct" = "green", "None" = "yellow", "Incorrect" = "red"), guide = guide_legend(reverse = TRUE))

Initial model

The type of the outcome is adjacent categories and is modeled as a cumulative family. TODO FIX!

We include high_debt_verison as well as a varying intercept for each individual in our initial model.

Selecting Priors

We iterate over the model until we have sane priors


documentation0.with <- extendable_model(
  base_name = "documentation0",
  base_formula = "documentation ~ 1 + high_debt_version + (1 | session)",
  base_priors = c(
    prior(normal(0, 0.9), class = "b"),
    prior(normal(0, 0.9), class = "Intercept"),
    prior(exponential(1), class = "sd", dpar = "muIncorrect"),
    prior(exponential(1), class = "sd", dpar = "muNone")
  ),
  family = categorical(),
  data = d.both_completed,
)

# Default priors:
prior_summary(documentation0.with(only_priors= TRUE))
                prior     class                   coef   group resp        dpar nlpar bound       source
               (flat)         b                                     muIncorrect                  default
               (flat)         b high_debt_versionfalse              muIncorrect             (vectorized)
               (flat)         b                                          muNone                  default
               (flat)         b high_debt_versionfalse                   muNone             (vectorized)
 student_t(3, 0, 2.5) Intercept                                     muIncorrect                  default
 student_t(3, 0, 2.5) Intercept                                          muNone                  default
 student_t(3, 0, 2.5)        sd                                     muIncorrect                  default
 student_t(3, 0, 2.5)        sd                                          muNone                  default
 student_t(3, 0, 2.5)        sd                        session      muIncorrect             (vectorized)
 student_t(3, 0, 2.5)        sd              Intercept session      muIncorrect             (vectorized)
 student_t(3, 0, 2.5)        sd                        session           muNone             (vectorized)
 student_t(3, 0, 2.5)        sd              Intercept session           muNone             (vectorized)
# Our priors:
prior_summary(documentation0.with(sample_prior = "only"))
Specifying global priors for regression coefficients in categorical models is deprecated. Please specify priors separately for each response category.Specifying global priors for regression coefficients in categorical models is deprecated. Please specify priors separately for each response category.
          prior     class                   coef   group resp        dpar nlpar bound       source
 normal(0, 0.9)         b                                     muIncorrect                  default
 normal(0, 0.9)         b high_debt_versionfalse              muIncorrect             (vectorized)
 normal(0, 0.9)         b                                          muNone                  default
 normal(0, 0.9)         b high_debt_versionfalse                   muNone             (vectorized)
 normal(0, 0.9) Intercept                                     muIncorrect                  default
 normal(0, 0.9) Intercept                                          muNone                  default
 exponential(1)        sd                                     muIncorrect                     user
 exponential(1)        sd                                          muNone                     user
 exponential(1)        sd                        session      muIncorrect             (vectorized)
 exponential(1)        sd              Intercept session      muIncorrect             (vectorized)
 exponential(1)        sd                        session           muNone             (vectorized)
 exponential(1)        sd              Intercept session           muNone             (vectorized)
# Prior predictive check
pp_check(documentation0.with(sample_prior = "only"), nsamples = 200, type = "bars")
Specifying global priors for regression coefficients in categorical models is deprecated. Please specify priors separately for each response category.Specifying global priors for regression coefficients in categorical models is deprecated. Please specify priors separately for each response category.

Model fit

We check the posterior distribution and can see that the model seems to have been able to fit the data well

# Posterior predictive check
pp_check(documentation0.with(), nsamples = 100, type = "bars")
Specifying global priors for regression coefficients in categorical models is deprecated. Please specify priors separately for each response category.Specifying global priors for regression coefficients in categorical models is deprecated. Please specify priors separately for each response category.

summary(documentation0.with())
Specifying global priors for regression coefficients in categorical models is deprecated. Please specify priors separately for each response category.Specifying global priors for regression coefficients in categorical models is deprecated. Please specify priors separately for each response category.
 Family: categorical 
  Links: muIncorrect = logit; muNone = logit 
Formula: documentation ~ 1 + high_debt_version + (1 | session) 
   Data: as.data.frame(data) (Number of observations: 44) 
Samples: 4 chains, each with iter = 2000; warmup = 1000; thin = 1;
         total post-warmup samples = 4000

Group-Level Effects: 
~session (Number of levels: 22) 
                          Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS Tail_ESS
sd(muIncorrect_Intercept)     1.32      0.97     0.06     3.56 1.01      675     1609
sd(muNone_Intercept)          1.51      1.07     0.04     3.99 1.01      496     1177

Population-Level Effects: 
                                   Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS Tail_ESS
muIncorrect_Intercept                  0.20      0.63    -1.13     1.33 1.00     3196     2478
muNone_Intercept                       0.78      0.59    -0.45     1.91 1.00     2713     2323
muIncorrect_high_debt_versionfalse    -0.07      0.65    -1.31     1.20 1.00     4366     2796
muNone_high_debt_versionfalse          0.30      0.62    -0.96     1.55 1.00     4690     2912

Samples were drawn using sample(hmc). For each parameter, Bulk_ESS
and Tail_ESS are effective sample size measures, and Rhat is the potential
scale reduction factor on split chains (at convergence, Rhat = 1).

Model extenstions

We use loo to check some possible extensions on the model.


edlvl_prior <- c(
  prior(dirichlet(2), class = "simo", coef = "moeducation_level1", dpar = "muIncorrect"),
  prior(dirichlet(2), class = "simo", coef = "moeducation_level1", dpar = "muNone")
)

loo(
  documentation0.with(),
  documentation0.with("work_domain"),
  documentation0.with("work_experience_programming.s"),
  documentation0.with("work_experience_java.s"),
  documentation0.with("education_field"),
  documentation0.with("mo(education_level)", edlvl_prior),
  documentation0.with("workplace_peer_review"),
  documentation0.with("workplace_td_tracking"),
  documentation0.with("workplace_pair_programming"),
  documentation0.with("workplace_coding_standards"),
  documentation0.with("scenario"),
  documentation0.with("group"),
  documentation0.with("order")
)
Specifying global priors for regression coefficients in categorical models is deprecated. Please specify priors separately for each response category.Specifying global priors for regression coefficients in categorical models is deprecated. Please specify priors separately for each response category.Specifying global priors for regression coefficients in categorical models is deprecated. Please specify priors separately for each response category.Specifying global priors for regression coefficients in categorical models is deprecated. Please specify priors separately for each response category.Specifying global priors for regression coefficients in categorical models is deprecated. Please specify priors separately for each response category.Specifying global priors for regression coefficients in categorical models is deprecated. Please specify priors separately for each response category.Specifying global priors for regression coefficients in categorical models is deprecated. Please specify priors separately for each response category.Specifying global priors for regression coefficients in categorical models is deprecated. Please specify priors separately for each response category.Specifying global priors for regression coefficients in categorical models is deprecated. Please specify priors separately for each response category.Specifying global priors for regression coefficients in categorical models is deprecated. Please specify priors separately for each response category.Specifying global priors for regression coefficients in categorical models is deprecated. Please specify priors separately for each response category.Specifying global priors for regression coefficients in categorical models is deprecated. Please specify priors separately for each response category.Specifying global priors for regression coefficients in categorical models is deprecated. Please specify priors separately for each response category.Specifying global priors for regression coefficients in categorical models is deprecated. Please specify priors separately for each response category.Specifying global priors for regression coefficients in categorical models is deprecated. Please specify priors separately for each response category.Specifying global priors for regression coefficients in categorical models is deprecated. Please specify priors separately for each response category.Specifying global priors for regression coefficients in categorical models is deprecated. Please specify priors separately for each response category.Specifying global priors for regression coefficients in categorical models is deprecated. Please specify priors separately for each response category.Specifying global priors for regression coefficients in categorical models is deprecated. Please specify priors separately for each response category.Specifying global priors for regression coefficients in categorical models is deprecated. Please specify priors separately for each response category.Specifying global priors for regression coefficients in categorical models is deprecated. Please specify priors separately for each response category.Specifying global priors for regression coefficients in categorical models is deprecated. Please specify priors separately for each response category.Specifying global priors for regression coefficients in categorical models is deprecated. Please specify priors separately for each response category.Specifying global priors for regression coefficients in categorical models is deprecated. Please specify priors separately for each response category.Specifying global priors for regression coefficients in categorical models is deprecated. Please specify priors separately for each response category.Specifying global priors for regression coefficients in categorical models is deprecated. Please specify priors separately for each response category.Found 2 observations with a pareto_k > 0.7 in model 'documentation0.with()'. It is recommended to set 'moment_match = TRUE' in order to perform moment matching for problematic observations. Found 6 observations with a pareto_k > 0.7 in model 'documentation0.with("work_domain")'. It is recommended to set 'moment_match = TRUE' in order to perform moment matching for problematic observations. Found 10 observations with a pareto_k > 0.7 in model 'documentation0.with("work_experience_programming.s")'. It is recommended to set 'moment_match = TRUE' in order to perform moment matching for problematic observations. Found 8 observations with a pareto_k > 0.7 in model 'documentation0.with("work_experience_java.s")'. It is recommended to set 'moment_match = TRUE' in order to perform moment matching for problematic observations. Found 4 observations with a pareto_k > 0.7 in model 'documentation0.with("education_field")'. It is recommended to set 'moment_match = TRUE' in order to perform moment matching for problematic observations. Found 3 observations with a pareto_k > 0.7 in model 'documentation0.with("mo(education_level)", edlvl_prior)'. It is recommended to set 'moment_match = TRUE' in order to perform moment matching for problematic observations. Found 6 observations with a pareto_k > 0.7 in model 'documentation0.with("workplace_peer_review")'. It is recommended to set 'moment_match = TRUE' in order to perform moment matching for problematic observations. Found 3 observations with a pareto_k > 0.7 in model 'documentation0.with("workplace_td_tracking")'. It is recommended to set 'moment_match = TRUE' in order to perform moment matching for problematic observations. Found 9 observations with a pareto_k > 0.7 in model 'documentation0.with("workplace_pair_programming")'. It is recommended to set 'moment_match = TRUE' in order to perform moment matching for problematic observations. Found 2 observations with a pareto_k > 0.7 in model 'documentation0.with("workplace_coding_standards")'. It is recommended to set 'moment_match = TRUE' in order to perform moment matching for problematic observations. Found 5 observations with a pareto_k > 0.7 in model 'documentation0.with("scenario")'. It is recommended to set 'moment_match = TRUE' in order to perform moment matching for problematic observations. Found 5 observations with a pareto_k > 0.7 in model 'documentation0.with("group")'. It is recommended to set 'moment_match = TRUE' in order to perform moment matching for problematic observations. Found 6 observations with a pareto_k > 0.7 in model 'documentation0.with("order")'. It is recommended to set 'moment_match = TRUE' in order to perform moment matching for problematic observations. 
Output of model 'documentation0.with()':

Computed from 4000 by 44 log-likelihood matrix
------
Monte Carlo SE of elpd_loo is NA.

Pareto k diagnostic values:
                         Count Pct.    Min. n_eff
(-Inf, 0.5]   (good)     25    56.8%   776       
 (0.5, 0.7]   (ok)       17    38.6%   424       
   (0.7, 1]   (bad)       2     4.5%   397       
   (1, Inf)   (very bad)  0     0.0%   <NA>      
See help('pareto-k-diagnostic') for details.

Output of model 'documentation0.with("work_domain")':

Computed from 4000 by 44 log-likelihood matrix
------
Monte Carlo SE of elpd_loo is NA.

Pareto k diagnostic values:
                         Count Pct.    Min. n_eff
(-Inf, 0.5]   (good)     26    59.1%   869       
 (0.5, 0.7]   (ok)       12    27.3%   415       
   (0.7, 1]   (bad)       6    13.6%   260       
   (1, Inf)   (very bad)  0     0.0%   <NA>      
See help('pareto-k-diagnostic') for details.

Output of model 'documentation0.with("work_experience_programming.s")':

Computed from 4000 by 44 log-likelihood matrix
------
Monte Carlo SE of elpd_loo is NA.

Pareto k diagnostic values:
                         Count Pct.    Min. n_eff
(-Inf, 0.5]   (good)     25    56.8%   999       
 (0.5, 0.7]   (ok)        9    20.5%   423       
   (0.7, 1]   (bad)      10    22.7%   141       
   (1, Inf)   (very bad)  0     0.0%   <NA>      
See help('pareto-k-diagnostic') for details.

Output of model 'documentation0.with("work_experience_java.s")':

Computed from 4000 by 44 log-likelihood matrix
------
Monte Carlo SE of elpd_loo is NA.

Pareto k diagnostic values:
                         Count Pct.    Min. n_eff
(-Inf, 0.5]   (good)     26    59.1%   955       
 (0.5, 0.7]   (ok)       10    22.7%   260       
   (0.7, 1]   (bad)       8    18.2%   127       
   (1, Inf)   (very bad)  0     0.0%   <NA>      
See help('pareto-k-diagnostic') for details.

Output of model 'documentation0.with("education_field")':

Computed from 4000 by 44 log-likelihood matrix
------
Monte Carlo SE of elpd_loo is NA.

Pareto k diagnostic values:
                         Count Pct.    Min. n_eff
(-Inf, 0.5]   (good)     26    59.1%   703       
 (0.5, 0.7]   (ok)       14    31.8%   256       
   (0.7, 1]   (bad)       4     9.1%   280       
   (1, Inf)   (very bad)  0     0.0%   <NA>      
See help('pareto-k-diagnostic') for details.

Output of model 'documentation0.with("mo(education_level)", edlvl_prior)':

Computed from 4000 by 44 log-likelihood matrix
------
Monte Carlo SE of elpd_loo is NA.

Pareto k diagnostic values:
                         Count Pct.    Min. n_eff
(-Inf, 0.5]   (good)     26    59.1%   697       
 (0.5, 0.7]   (ok)       15    34.1%   485       
   (0.7, 1]   (bad)       3     6.8%   172       
   (1, Inf)   (very bad)  0     0.0%   <NA>      
See help('pareto-k-diagnostic') for details.

Output of model 'documentation0.with("workplace_peer_review")':

Computed from 4000 by 44 log-likelihood matrix
------
Monte Carlo SE of elpd_loo is NA.

Pareto k diagnostic values:
                         Count Pct.    Min. n_eff
(-Inf, 0.5]   (good)     26    59.1%   1070      
 (0.5, 0.7]   (ok)       12    27.3%   473       
   (0.7, 1]   (bad)       6    13.6%   176       
   (1, Inf)   (very bad)  0     0.0%   <NA>      
See help('pareto-k-diagnostic') for details.

Output of model 'documentation0.with("workplace_td_tracking")':

Computed from 4000 by 44 log-likelihood matrix
------
Monte Carlo SE of elpd_loo is NA.

Pareto k diagnostic values:
                         Count Pct.    Min. n_eff
(-Inf, 0.5]   (good)     21    47.7%   1031      
 (0.5, 0.7]   (ok)       20    45.5%   214       
   (0.7, 1]   (bad)       3     6.8%   262       
   (1, Inf)   (very bad)  0     0.0%   <NA>      
See help('pareto-k-diagnostic') for details.

Output of model 'documentation0.with("workplace_pair_programming")':

Computed from 4000 by 44 log-likelihood matrix
------
Monte Carlo SE of elpd_loo is NA.

Pareto k diagnostic values:
                         Count Pct.    Min. n_eff
(-Inf, 0.5]   (good)     23    52.3%   946       
 (0.5, 0.7]   (ok)       12    27.3%   496       
   (0.7, 1]   (bad)       9    20.5%   155       
   (1, Inf)   (very bad)  0     0.0%   <NA>      
See help('pareto-k-diagnostic') for details.

Output of model 'documentation0.with("workplace_coding_standards")':

Computed from 4000 by 44 log-likelihood matrix
------
Monte Carlo SE of elpd_loo is NA.

Pareto k diagnostic values:
                         Count Pct.    Min. n_eff
(-Inf, 0.5]   (good)     25    56.8%   580       
 (0.5, 0.7]   (ok)       17    38.6%   318       
   (0.7, 1]   (bad)       2     4.5%   529       
   (1, Inf)   (very bad)  0     0.0%   <NA>      
See help('pareto-k-diagnostic') for details.

Output of model 'documentation0.with("scenario")':

Computed from 4000 by 44 log-likelihood matrix
------
Monte Carlo SE of elpd_loo is NA.

Pareto k diagnostic values:
                         Count Pct.    Min. n_eff
(-Inf, 0.5]   (good)     21    47.7%   541       
 (0.5, 0.7]   (ok)       18    40.9%   253       
   (0.7, 1]   (bad)       5    11.4%   297       
   (1, Inf)   (very bad)  0     0.0%   <NA>      
See help('pareto-k-diagnostic') for details.

Output of model 'documentation0.with("group")':

Computed from 4000 by 44 log-likelihood matrix
------
Monte Carlo SE of elpd_loo is NA.

Pareto k diagnostic values:
                         Count Pct.    Min. n_eff
(-Inf, 0.5]   (good)     28    63.6%   464       
 (0.5, 0.7]   (ok)       11    25.0%   271       
   (0.7, 1]   (bad)       5    11.4%   252       
   (1, Inf)   (very bad)  0     0.0%   <NA>      
See help('pareto-k-diagnostic') for details.

Output of model 'documentation0.with("order")':

Computed from 4000 by 44 log-likelihood matrix
------
Monte Carlo SE of elpd_loo is NA.

Pareto k diagnostic values:
                         Count Pct.    Min. n_eff
(-Inf, 0.5]   (good)     21    47.7%   873       
 (0.5, 0.7]   (ok)       17    38.6%   202       
   (0.7, 1]   (bad)       6    13.6%   114       
   (1, Inf)   (very bad)  0     0.0%   <NA>      
See help('pareto-k-diagnostic') for details.

Model comparisons:
                                                        elpd_diff se_diff
documentation0.with("workplace_peer_review")             0.0       0.0   
documentation0.with("work_domain")                      -0.2       1.2   
documentation0.with()                                   -0.4       0.7   
documentation0.with("workplace_td_tracking")            -0.6       1.1   
documentation0.with("workplace_coding_standards")       -0.7       0.6   
documentation0.with("education_field")                  -0.7       0.9   
documentation0.with("group")                            -0.7       0.9   
documentation0.with("workplace_pair_programming")       -1.1       0.9   
documentation0.with("mo(education_level)", edlvl_prior) -1.1       1.0   
documentation0.with("work_experience_java.s")           -1.2       1.1   
documentation0.with("order")                            -1.3       1.5   
documentation0.with("scenario")                         -1.5       0.9   
documentation0.with("work_experience_programming.s")    -1.5       1.1   

We pick out some interesting variables and try combining them

loo(
  documentation0.with(),
  documentation0.with("work_domain"),
  documentation0.with("workplace_peer_review"),
  documentation0.with("workplace_td_tracking"),
  
  documentation0.with(c("work_domain", "workplace_peer_review")),
  documentation0.with(c("work_domain", "workplace_td_tracking")),
  
  documentation0.with(c("workplace_peer_review", "workplace_td_tracking")),
  
  documentation0.with(c("work_domain", "workplace_peer_review", "workplace_td_tracking"))
)
Specifying global priors for regression coefficients in categorical models is deprecated. Please specify priors separately for each response category.Specifying global priors for regression coefficients in categorical models is deprecated. Please specify priors separately for each response category.Specifying global priors for regression coefficients in categorical models is deprecated. Please specify priors separately for each response category.Specifying global priors for regression coefficients in categorical models is deprecated. Please specify priors separately for each response category.Specifying global priors for regression coefficients in categorical models is deprecated. Please specify priors separately for each response category.Specifying global priors for regression coefficients in categorical models is deprecated. Please specify priors separately for each response category.Specifying global priors for regression coefficients in categorical models is deprecated. Please specify priors separately for each response category.Specifying global priors for regression coefficients in categorical models is deprecated. Please specify priors separately for each response category.Specifying global priors for regression coefficients in categorical models is deprecated. Please specify priors separately for each response category.Specifying global priors for regression coefficients in categorical models is deprecated. Please specify priors separately for each response category.Specifying global priors for regression coefficients in categorical models is deprecated. Please specify priors separately for each response category.Specifying global priors for regression coefficients in categorical models is deprecated. Please specify priors separately for each response category.Specifying global priors for regression coefficients in categorical models is deprecated. Please specify priors separately for each response category.Specifying global priors for regression coefficients in categorical models is deprecated. Please specify priors separately for each response category.Specifying global priors for regression coefficients in categorical models is deprecated. Please specify priors separately for each response category.Specifying global priors for regression coefficients in categorical models is deprecated. Please specify priors separately for each response category.Found 2 observations with a pareto_k > 0.7 in model 'documentation0.with()'. It is recommended to set 'moment_match = TRUE' in order to perform moment matching for problematic observations. Found 6 observations with a pareto_k > 0.7 in model 'documentation0.with("work_domain")'. It is recommended to set 'moment_match = TRUE' in order to perform moment matching for problematic observations. Found 6 observations with a pareto_k > 0.7 in model 'documentation0.with("workplace_peer_review")'. It is recommended to set 'moment_match = TRUE' in order to perform moment matching for problematic observations. Found 3 observations with a pareto_k > 0.7 in model 'documentation0.with("workplace_td_tracking")'. It is recommended to set 'moment_match = TRUE' in order to perform moment matching for problematic observations. Found 2 observations with a pareto_k > 0.7 in model 'documentation0.with(c("work_domain", "workplace_peer_review"))'. It is recommended to set 'moment_match = TRUE' in order to perform moment matching for problematic observations. Found 1 observations with a pareto_k > 0.7 in model 'documentation0.with(c("work_domain", "workplace_td_tracking"))'. It is recommended to set 'moment_match = TRUE' in order to perform moment matching for problematic observations. Found 4 observations with a pareto_k > 0.7 in model 'documentation0.with(c("workplace_peer_review", "workplace_td_tracking"))'. It is recommended to set 'moment_match = TRUE' in order to perform moment matching for problematic observations. Found 4 observations with a pareto_k > 0.7 in model 'documentation0.with(c("work_domain", "workplace_peer_review",     "workplace_td_tracking"))'. It is recommended to set 'moment_match = TRUE' in order to perform moment matching for problematic observations. 
Output of model 'documentation0.with()':

Computed from 4000 by 44 log-likelihood matrix
------
Monte Carlo SE of elpd_loo is NA.

Pareto k diagnostic values:
                         Count Pct.    Min. n_eff
(-Inf, 0.5]   (good)     25    56.8%   776       
 (0.5, 0.7]   (ok)       17    38.6%   424       
   (0.7, 1]   (bad)       2     4.5%   397       
   (1, Inf)   (very bad)  0     0.0%   <NA>      
See help('pareto-k-diagnostic') for details.

Output of model 'documentation0.with("work_domain")':

Computed from 4000 by 44 log-likelihood matrix
------
Monte Carlo SE of elpd_loo is NA.

Pareto k diagnostic values:
                         Count Pct.    Min. n_eff
(-Inf, 0.5]   (good)     26    59.1%   869       
 (0.5, 0.7]   (ok)       12    27.3%   415       
   (0.7, 1]   (bad)       6    13.6%   260       
   (1, Inf)   (very bad)  0     0.0%   <NA>      
See help('pareto-k-diagnostic') for details.

Output of model 'documentation0.with("workplace_peer_review")':

Computed from 4000 by 44 log-likelihood matrix
------
Monte Carlo SE of elpd_loo is NA.

Pareto k diagnostic values:
                         Count Pct.    Min. n_eff
(-Inf, 0.5]   (good)     26    59.1%   1070      
 (0.5, 0.7]   (ok)       12    27.3%   473       
   (0.7, 1]   (bad)       6    13.6%   176       
   (1, Inf)   (very bad)  0     0.0%   <NA>      
See help('pareto-k-diagnostic') for details.

Output of model 'documentation0.with("workplace_td_tracking")':

Computed from 4000 by 44 log-likelihood matrix
------
Monte Carlo SE of elpd_loo is NA.

Pareto k diagnostic values:
                         Count Pct.    Min. n_eff
(-Inf, 0.5]   (good)     21    47.7%   1031      
 (0.5, 0.7]   (ok)       20    45.5%   214       
   (0.7, 1]   (bad)       3     6.8%   262       
   (1, Inf)   (very bad)  0     0.0%   <NA>      
See help('pareto-k-diagnostic') for details.

Output of model 'documentation0.with(c("work_domain", "workplace_peer_review"))':

Computed from 4000 by 44 log-likelihood matrix
------
Monte Carlo SE of elpd_loo is NA.

Pareto k diagnostic values:
                         Count Pct.    Min. n_eff
(-Inf, 0.5]   (good)     28    63.6%   877       
 (0.5, 0.7]   (ok)       14    31.8%   440       
   (0.7, 1]   (bad)       2     4.5%   334       
   (1, Inf)   (very bad)  0     0.0%   <NA>      
See help('pareto-k-diagnostic') for details.

Output of model 'documentation0.with(c("work_domain", "workplace_td_tracking"))':

Computed from 4000 by 44 log-likelihood matrix
------
Monte Carlo SE of elpd_loo is NA.

Pareto k diagnostic values:
                         Count Pct.    Min. n_eff
(-Inf, 0.5]   (good)     28    63.6%   797       
 (0.5, 0.7]   (ok)       15    34.1%   492       
   (0.7, 1]   (bad)       1     2.3%   324       
   (1, Inf)   (very bad)  0     0.0%   <NA>      
See help('pareto-k-diagnostic') for details.

Output of model 'documentation0.with(c("workplace_peer_review", "workplace_td_tracking"))':

Computed from 4000 by 44 log-likelihood matrix
------
Monte Carlo SE of elpd_loo is NA.

Pareto k diagnostic values:
                         Count Pct.    Min. n_eff
(-Inf, 0.5]   (good)     25    56.8%   957       
 (0.5, 0.7]   (ok)       15    34.1%   270       
   (0.7, 1]   (bad)       4     9.1%   187       
   (1, Inf)   (very bad)  0     0.0%   <NA>      
See help('pareto-k-diagnostic') for details.

Output of model 'documentation0.with(c("work_domain", "workplace_peer_review",     "workplace_td_tracking"))':

Computed from 4000 by 44 log-likelihood matrix
------
Monte Carlo SE of elpd_loo is NA.

Pareto k diagnostic values:
                         Count Pct.    Min. n_eff
(-Inf, 0.5]   (good)     29    65.9%   1023      
 (0.5, 0.7]   (ok)       11    25.0%   675       
   (0.7, 1]   (bad)       4     9.1%   245       
   (1, Inf)   (very bad)  0     0.0%   <NA>      
See help('pareto-k-diagnostic') for details.

Model comparisons:
                                                                                            elpd_diff se_diff
documentation0.with(c("work_domain", "workplace_peer_review",     "workplace_td_tracking"))  0.0       0.0   
documentation0.with(c("work_domain", "workplace_peer_review"))                              -0.4       0.7   
documentation0.with(c("work_domain", "workplace_td_tracking"))                              -0.9       0.8   
documentation0.with("workplace_peer_review")                                                -1.0       1.0   
documentation0.with(c("workplace_peer_review", "workplace_td_tracking"))                    -1.2       0.9   
documentation0.with("work_domain")                                                          -1.2       0.8   
documentation0.with()                                                                       -1.4       1.0   
documentation0.with("workplace_td_tracking")                                                -1.5       0.9   

Candidate models

We inspect some of our top performing models.

All models seems to have sampled nicely (rhat is ca 1 and fluffy plots) they also have about the same fit to the data end similar estimated for the high_debt_version beta parameter

documentation0 <- documentation0.with()
Specifying global priors for regression coefficients in categorical models is deprecated. Please specify priors separately for each response category.Specifying global priors for regression coefficients in categorical models is deprecated. Please specify priors separately for each response category.
summary(documentation0)
 Family: categorical 
  Links: muIncorrect = logit; muNone = logit 
Formula: documentation ~ 1 + high_debt_version + (1 | session) 
   Data: as.data.frame(data) (Number of observations: 44) 
Samples: 4 chains, each with iter = 2000; warmup = 1000; thin = 1;
         total post-warmup samples = 4000

Group-Level Effects: 
~session (Number of levels: 22) 
                          Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS Tail_ESS
sd(muIncorrect_Intercept)     1.32      0.97     0.06     3.56 1.01      675     1609
sd(muNone_Intercept)          1.51      1.07     0.04     3.99 1.01      496     1177

Population-Level Effects: 
                                   Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS Tail_ESS
muIncorrect_Intercept                  0.20      0.63    -1.13     1.33 1.00     3196     2478
muNone_Intercept                       0.78      0.59    -0.45     1.91 1.00     2713     2323
muIncorrect_high_debt_versionfalse    -0.07      0.65    -1.31     1.20 1.00     4366     2796
muNone_high_debt_versionfalse          0.30      0.62    -0.96     1.55 1.00     4690     2912

Samples were drawn using sample(hmc). For each parameter, Bulk_ESS
and Tail_ESS are effective sample size measures, and Rhat is the potential
scale reduction factor on split chains (at convergence, Rhat = 1).
ranef(documentation0)
$session
, , muIncorrect_Intercept

                           Estimate Est.Error       Q2.5    Q97.5
6033d69a5af2c702367b3a95 -0.7470067  1.444398 -4.4761393 1.213727
6033d90a5af2c702367b3a96 -0.5784250  1.431037 -4.2015125 1.690868
6034fc165af2c702367b3a98 -0.5797671  1.491558 -4.4219490 1.817742
603500725af2c702367b3a99  1.4013453  1.661203 -0.5322208 5.435977
603f97625af2c702367b3a9d  1.3861995  1.727238 -0.5869947 5.880972
603fd5d95af2c702367b3a9e -0.5532681  1.499100 -4.2581675 1.768124
60409b7b5af2c702367b3a9f  1.3688257  1.716254 -0.6373885 5.764173
604b82b5a7718fbed181b336 -0.5383897  1.433520 -4.0766335 1.681776
6050c1bf856f36729d2e5218  1.3915309  1.708214 -0.5543257 5.785509
6050e1e7856f36729d2e5219  0.2889439  1.086331 -1.7661648 2.821988
6055fdc6856f36729d2e521b -0.5521420  1.400777 -4.0275050 1.594849
60589862856f36729d2e521f  0.3169461  1.105256 -1.8033922 2.901520
605a30a7856f36729d2e5221 -0.7225399  1.475082 -4.5018520 1.407789
605afa3a856f36729d2e5222 -0.7105838  1.450999 -4.5794740 1.340559
605c8bc6856f36729d2e5223  0.5291758  1.167988 -1.4395825 3.403833
605f3f2d856f36729d2e5224 -0.5483810  1.444796 -4.0999982 1.694556
605f46c3856f36729d2e5225 -0.5629205  1.471839 -4.1716535 1.856721
60605337856f36729d2e5226  0.3190511  1.092574 -1.6690627 2.885536
60609ae6856f36729d2e5228  0.5281284  1.125644 -1.3691678 3.230003
6061ce91856f36729d2e522e -0.7383476  1.468647 -4.5413172 1.343341
6061f106856f36729d2e5231 -0.5875646  1.473553 -4.4078757 1.661198
6068ea9f856f36729d2e523e -0.5777273  1.450281 -4.0351175 1.629520

, , muNone_Intercept

                            Estimate Est.Error       Q2.5     Q97.5
6033d69a5af2c702367b3a95 -0.05558621  1.118638 -2.4188793 2.3273868
6033d90a5af2c702367b3a96  1.24308232  1.708663 -0.9343491 5.7670788
6034fc165af2c702367b3a98  1.23147343  1.681031 -0.8807831 5.4403865
603500725af2c702367b3a99 -1.06045831  1.653527 -5.2565720 1.1949475
603f97625af2c702367b3a9d -1.02688063  1.688710 -5.3299650 1.3145548
603fd5d95af2c702367b3a9e  1.25282665  1.703018 -0.8742166 5.8370990
60409b7b5af2c702367b3a9f -1.03831960  1.667494 -5.3294795 1.0947363
604b82b5a7718fbed181b336  1.27507181  1.738235 -0.8283333 5.5911513
6050c1bf856f36729d2e5218 -0.99799157  1.650894 -5.1530970 1.3357060
6050e1e7856f36729d2e5219 -1.20729147  1.647126 -5.3844693 0.8409788
6055fdc6856f36729d2e521b  1.26583853  1.693720 -0.8129744 5.5921383
60589862856f36729d2e521f -1.17927492  1.644214 -5.3838805 0.9207417
605a30a7856f36729d2e5221 -0.04796134  1.094246 -2.3835817 2.2673958
605afa3a856f36729d2e5222 -0.03272103  1.087160 -2.3258822 2.2620213
605c8bc6856f36729d2e5223  0.13602894  1.157270 -2.2280525 2.6387448
605f3f2d856f36729d2e5224  1.27705082  1.745486 -0.8647861 5.6147463
605f46c3856f36729d2e5225  1.25360191  1.713079 -0.8028035 5.8153710
60605337856f36729d2e5226 -1.22352306  1.622208 -5.2776060 0.8247435
60609ae6856f36729d2e5228  0.12778528  1.151828 -2.1453255 2.6740225
6061ce91856f36729d2e522e -0.07847170  1.066914 -2.2713707 2.1473563
6061f106856f36729d2e5231  1.22218268  1.657472 -0.9209316 5.5727133
6068ea9f856f36729d2e523e  1.24733828  1.676535 -0.8875774 5.4576228
plot(documentation0, ask = FALSE)


pp_check(documentation0, nsamples = 150, type= "bars")

documentation1 <- documentation0.with("workplace_peer_review")
Specifying global priors for regression coefficients in categorical models is deprecated. Please specify priors separately for each response category.Specifying global priors for regression coefficients in categorical models is deprecated. Please specify priors separately for each response category.
summary(documentation1)
 Family: categorical 
  Links: muIncorrect = logit; muNone = logit 
Formula: documentation ~ 1 + high_debt_version + (1 | session) + workplace_peer_review 
   Data: as.data.frame(data) (Number of observations: 44) 
Samples: 4 chains, each with iter = 2000; warmup = 1000; thin = 1;
         total post-warmup samples = 4000

Group-Level Effects: 
~session (Number of levels: 22) 
                          Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS Tail_ESS
sd(muIncorrect_Intercept)     1.56      1.05     0.08     3.91 1.00      655     1558
sd(muNone_Intercept)          1.29      1.02     0.04     3.76 1.00      673     1650

Population-Level Effects: 
                                       Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS Tail_ESS
muIncorrect_Intercept                      0.15      0.70    -1.31     1.49 1.00     3358     2562
muNone_Intercept                           0.46      0.62    -0.80     1.72 1.00     3645     2789
muIncorrect_high_debt_versionfalse        -0.09      0.66    -1.39     1.21 1.00     5105     2898
muIncorrect_workplace_peer_reviewfalse     0.04      0.74    -1.45     1.45 1.00     3753     2715
muNone_high_debt_versionfalse              0.30      0.62    -0.89     1.54 1.00     4307     2926
muNone_workplace_peer_reviewfalse          0.81      0.68    -0.55     2.10 1.00     3795     3050

Samples were drawn using sample(hmc). For each parameter, Bulk_ESS
and Tail_ESS are effective sample size measures, and Rhat is the potential
scale reduction factor on split chains (at convergence, Rhat = 1).
ranef(documentation1)
$session
, , muIncorrect_Intercept

                           Estimate Est.Error       Q2.5    Q97.5
6033d69a5af2c702367b3a95 -0.7796731  1.555826 -4.7689282 1.635552
6033d90a5af2c702367b3a96 -0.6479978  1.581244 -4.5662600 1.843063
6034fc165af2c702367b3a98 -0.6813477  1.563078 -4.5821568 1.810310
603500725af2c702367b3a99  1.8350614  1.932535 -0.5654696 6.669127
603f97625af2c702367b3a9d  1.6808429  1.878981 -0.5818434 6.375892
603fd5d95af2c702367b3a9e -0.6639747  1.694110 -5.0239417 1.961679
60409b7b5af2c702367b3a9f  1.8157182  1.919123 -0.5260058 6.436726
604b82b5a7718fbed181b336 -0.6596849  1.642309 -4.7123495 1.956744
6050c1bf856f36729d2e5218  1.6666907  1.806904 -0.5922459 6.089604
6050e1e7856f36729d2e5219  0.3388241  1.179005 -1.8741260 2.946523
6055fdc6856f36729d2e521b -0.7868763  1.659068 -5.1004890 1.720580
60589862856f36729d2e521f  0.3627940  1.199089 -1.9899422 3.035509
605a30a7856f36729d2e5221 -0.9173599  1.566438 -4.9564925 1.319932
605afa3a856f36729d2e5222 -0.9515436  1.640601 -5.0961575 1.424776
605c8bc6856f36729d2e5223  0.5588737  1.233204 -1.6452240 3.379121
605f3f2d856f36729d2e5224 -0.7915956  1.687693 -5.2219490 1.652847
605f46c3856f36729d2e5225 -0.7949234  1.650671 -5.1311007 1.708503
60605337856f36729d2e5226  0.3235350  1.190782 -2.0464560 2.976035
60609ae6856f36729d2e5228  0.7649240  1.330889 -1.3516427 3.888544
6061ce91856f36729d2e522e -0.9571343  1.617258 -5.1593610 1.255922
6061f106856f36729d2e5231 -0.6561428  1.611278 -4.6556502 1.855628
6068ea9f856f36729d2e523e -0.6546241  1.637685 -4.6738263 1.969629

, , muNone_Intercept

                             Estimate Est.Error       Q2.5     Q97.5
6033d69a5af2c702367b3a95 -0.172876265  1.062679 -2.5141473 1.9823890
6033d90a5af2c702367b3a96  0.915285084  1.617227 -1.0179812 5.2208248
6034fc165af2c702367b3a98  0.904656562  1.539997 -1.0547840 5.0003953
603500725af2c702367b3a99 -0.887607319  1.516669 -4.8790233 1.1131557
603f97625af2c702367b3a9d -0.742023507  1.549659 -4.6616405 1.4136495
603fd5d95af2c702367b3a9e  0.905783634  1.575857 -1.0669485 4.9609140
60409b7b5af2c702367b3a9f -0.879110738  1.498599 -4.6937335 1.0903668
604b82b5a7718fbed181b336  0.920801477  1.564911 -1.0394685 5.1456170
6050c1bf856f36729d2e5218 -0.754435881  1.551021 -4.6688002 1.3040945
6050e1e7856f36729d2e5219 -0.942469933  1.530720 -5.0538467 0.9844937
6055fdc6856f36729d2e521b  1.104470666  1.557338 -0.7118958 5.1568310
60589862856f36729d2e521f -0.911326288  1.483166 -4.7708107 0.9305530
605a30a7856f36729d2e5221  0.032161305  1.010984 -2.1122598 2.2793915
605afa3a856f36729d2e5222  0.045549452  1.023600 -2.0464425 2.3758812
605c8bc6856f36729d2e5223  0.213486085  1.064093 -1.8354748 2.7442638
605f3f2d856f36729d2e5224  1.115918637  1.619768 -0.7700016 5.3101545
605f46c3856f36729d2e5225  1.080125952  1.578760 -0.7559829 5.0225058
60605337856f36729d2e5226 -0.935572646  1.542393 -4.9690372 0.9980503
60609ae6856f36729d2e5228  0.003317732  1.041522 -2.1978980 2.3569345
6061ce91856f36729d2e522e  0.011013405  1.004339 -2.0909662 2.1890850
6061f106856f36729d2e5231  0.890725275  1.513538 -1.0443862 4.9602315
6068ea9f856f36729d2e523e  0.908896633  1.558264 -1.0533830 5.1896355
plot(documentation1, ask = FALSE)


pp_check(documentation1, nsamples = 150, type= "bars")

documentation2 <- documentation0.with(c("workplace_peer_review", "work_domain"))
Specifying global priors for regression coefficients in categorical models is deprecated. Please specify priors separately for each response category.Specifying global priors for regression coefficients in categorical models is deprecated. Please specify priors separately for each response category.
summary(documentation2)
 Family: categorical 
  Links: muIncorrect = logit; muNone = logit 
Formula: documentation ~ 1 + high_debt_version + (1 | session) + work_domain + workplace_peer_review 
   Data: as.data.frame(data) (Number of observations: 44) 
Samples: 4 chains, each with iter = 2000; warmup = 1000; thin = 1;
         total post-warmup samples = 4000

Group-Level Effects: 
~session (Number of levels: 22) 
                          Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS Tail_ESS
sd(muIncorrect_Intercept)     1.23      0.97     0.02     3.57 1.00      919     1432
sd(muNone_Intercept)          1.19      0.97     0.04     3.54 1.00      789     1956

Population-Level Effects: 
                                       Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS Tail_ESS
muIncorrect_Intercept                      0.14      0.70    -1.29     1.51 1.00     4141     2983
muNone_Intercept                           0.54      0.68    -0.79     1.88 1.00     3904     2907
muIncorrect_high_debt_versionfalse        -0.07      0.64    -1.33     1.18 1.00     5233     2898
muIncorrect_work_domainDevops              0.56      0.84    -1.09     2.23 1.00     4885     2738
muIncorrect_work_domainEmbedded            0.10      0.82    -1.48     1.67 1.00     5025     3010
muIncorrect_work_domainFinance            -0.53      0.82    -2.18     1.09 1.00     4234     2849
muIncorrect_work_domainMixed               0.06      0.83    -1.55     1.74 1.00     5398     3006
muIncorrect_work_domainMusic               0.20      0.83    -1.42     1.83 1.00     5311     3096
muIncorrect_work_domainNone                0.96      0.79    -0.62     2.51 1.00     3932     2869
muIncorrect_work_domainRetail             -0.14      0.85    -1.82     1.54 1.00     4728     2873
muIncorrect_work_domainTelecom            -0.22      0.86    -1.89     1.46 1.00     4760     3036
muIncorrect_work_domainWeb                -0.22      0.77    -1.71     1.27 1.00     5523     3074
muIncorrect_workplace_peer_reviewfalse    -0.13      0.71    -1.53     1.27 1.00     4800     3081
muNone_high_debt_versionfalse              0.33      0.61    -0.86     1.53 1.00     5150     3136
muNone_work_domainDevops                  -0.32      0.84    -1.98     1.33 1.00     4283     2534
muNone_work_domainEmbedded                -0.46      0.84    -2.07     1.21 1.00     4825     2904
muNone_work_domainFinance                 -0.28      0.79    -1.82     1.23 1.00     4672     3358
muNone_work_domainMixed                   -0.46      0.86    -2.12     1.25 1.00     4263     2263
muNone_work_domainMusic                    0.06      0.84    -1.54     1.69 1.00     5427     2961
muNone_work_domainNone                    -0.36      0.77    -1.84     1.16 1.00     4044     2884
muNone_work_domainRetail                   0.28      0.87    -1.39     1.96 1.00     4580     2720
muNone_work_domainTelecom                  0.41      0.83    -1.20     2.00 1.00     4867     3193
muNone_work_domainWeb                      0.02      0.72    -1.36     1.44 1.00     4756     2712
muNone_workplace_peer_reviewfalse          0.91      0.70    -0.51     2.25 1.00     3915     2584

Samples were drawn using sample(hmc). For each parameter, Bulk_ESS
and Tail_ESS are effective sample size measures, and Rhat is the potential
scale reduction factor on split chains (at convergence, Rhat = 1).
ranef(documentation2)
$session
, , muIncorrect_Intercept

                           Estimate Est.Error       Q2.5    Q97.5
6033d69a5af2c702367b3a95 -0.5030895  1.374476 -3.9307447 1.769319
6033d90a5af2c702367b3a96 -0.4158952  1.357298 -3.9386175 1.864107
6034fc165af2c702367b3a98 -0.4093931  1.403509 -4.0887083 1.982395
603500725af2c702367b3a99  1.1780586  1.650336 -0.7426813 5.397898
603f97625af2c702367b3a9d  1.3463367  1.653553 -0.5051912 5.683918
603fd5d95af2c702367b3a9e -0.4511179  1.426560 -4.0300525 1.744010
60409b7b5af2c702367b3a9f  1.2075002  1.730038 -0.7305759 5.688347
604b82b5a7718fbed181b336 -0.4179851  1.365920 -3.9708575 1.862799
6050c1bf856f36729d2e5218  1.1310149  1.667002 -0.7899115 5.510719
6050e1e7856f36729d2e5219  0.2272325  1.047187 -1.8139178 2.718641
6055fdc6856f36729d2e521b -0.5049231  1.422157 -4.0931628 1.836192
60589862856f36729d2e521f  0.1832636  1.056718 -1.9893550 2.630138
605a30a7856f36729d2e5221 -0.6233837  1.392982 -4.3186890 1.474438
605afa3a856f36729d2e5222 -0.6292202  1.376357 -4.3781348 1.388346
605c8bc6856f36729d2e5223  0.3887177  1.121497 -1.6174260 3.033014
605f3f2d856f36729d2e5224 -0.5463267  1.331255 -3.9673140 1.487726
605f46c3856f36729d2e5225 -0.5564023  1.355038 -3.9858710 1.550545
60605337856f36729d2e5226  0.2891333  1.064092 -1.7398563 2.774680
60609ae6856f36729d2e5228  0.3382981  1.149913 -1.7230827 3.150140
6061ce91856f36729d2e522e -0.6233638  1.467581 -4.3315405 1.417251
6061f106856f36729d2e5231 -0.6354975  1.386346 -4.1724217 1.518698
6068ea9f856f36729d2e523e -0.4164789  1.401642 -3.9128615 1.872550

, , muNone_Intercept

                            Estimate Est.Error       Q2.5     Q97.5
6033d69a5af2c702367b3a95 -0.24695357  1.057056 -2.6432085 1.9416808
6033d90a5af2c702367b3a96  0.76402720  1.476513 -1.1382385 4.6813198
6034fc165af2c702367b3a98  0.74398084  1.447418 -1.0958607 4.6706018
603500725af2c702367b3a99 -0.71170659  1.415355 -4.4113902 1.1574823
603f97625af2c702367b3a9d -0.78746011  1.484336 -4.6803907 1.1457588
603fd5d95af2c702367b3a9e  0.76024499  1.371167 -0.9772659 4.5589980
60409b7b5af2c702367b3a9f -0.71559812  1.396389 -4.3269297 1.1831625
604b82b5a7718fbed181b336  0.72385145  1.400408 -1.1500303 4.5489147
6050c1bf856f36729d2e5218 -0.61485084  1.421435 -4.1616568 1.5695170
6050e1e7856f36729d2e5219 -0.77352130  1.445093 -4.5717207 1.1377753
6055fdc6856f36729d2e521b  0.90884672  1.505380 -0.8858834 4.9570738
60589862856f36729d2e521f -0.78531000  1.464251 -4.5601387 1.1346605
605a30a7856f36729d2e5221 -0.04501468  1.029336 -2.2631813 2.0938388
605afa3a856f36729d2e5222  0.02082146  1.006806 -2.0929242 2.3135298
605c8bc6856f36729d2e5223  0.14621432  1.083157 -2.0527870 2.6032780
605f3f2d856f36729d2e5224  0.98109241  1.482548 -0.7751370 5.0230745
605f46c3856f36729d2e5225  0.98534284  1.475205 -0.7205648 4.6993383
60605337856f36729d2e5226 -0.87094825  1.405642 -4.5450395 0.9272302
60609ae6856f36729d2e5228  0.15423182  1.021028 -1.8388072 2.6485820
6061ce91856f36729d2e522e -0.01613100  1.006420 -2.1375935 2.3118230
6061f106856f36729d2e5231  0.98003671  1.558823 -0.9379544 5.1424430
6068ea9f856f36729d2e523e  0.73797729  1.520678 -1.2018937 4.5241027
plot(documentation2, ask = FALSE)


pp_check(documentation2, nsamples = 150, type= "bars")

documentation3 <- documentation0.with(c("workplace_peer_review", "work_domain", "workplace_td_tracking"))
Specifying global priors for regression coefficients in categorical models is deprecated. Please specify priors separately for each response category.Specifying global priors for regression coefficients in categorical models is deprecated. Please specify priors separately for each response category.
summary(documentation3)
 Family: categorical 
  Links: muIncorrect = logit; muNone = logit 
Formula: documentation ~ 1 + high_debt_version + (1 | session) + work_domain + workplace_peer_review + workplace_td_tracking 
   Data: as.data.frame(data) (Number of observations: 44) 
Samples: 4 chains, each with iter = 2000; warmup = 1000; thin = 1;
         total post-warmup samples = 4000

Group-Level Effects: 
~session (Number of levels: 22) 
                          Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS Tail_ESS
sd(muIncorrect_Intercept)     1.20      0.93     0.04     3.44 1.00      910     1621
sd(muNone_Intercept)          1.13      0.95     0.03     3.47 1.00      661     1776

Population-Level Effects: 
                                       Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS Tail_ESS
muIncorrect_Intercept                     -0.14      0.87    -1.88     1.54 1.00     3984     3109
muNone_Intercept                           0.92      0.81    -0.70     2.52 1.00     4890     3064
muIncorrect_high_debt_versionfalse        -0.09      0.63    -1.32     1.11 1.00     5865     3235
muIncorrect_work_domainDevops              0.53      0.85    -1.15     2.20 1.00     5481     3312
muIncorrect_work_domainEmbedded            0.16      0.83    -1.46     1.80 1.00     6201     3156
muIncorrect_work_domainFinance            -0.49      0.82    -2.13     1.08 1.00     5218     3204
muIncorrect_work_domainMixed               0.05      0.82    -1.60     1.68 1.00     5359     2995
muIncorrect_work_domainMusic               0.27      0.84    -1.38     1.92 1.00     5048     2756
muIncorrect_work_domainNone                0.90      0.77    -0.64     2.41 1.00     4973     2988
muIncorrect_work_domainRetail             -0.13      0.87    -1.87     1.61 1.00     6895     3000
muIncorrect_work_domainTelecom            -0.18      0.86    -1.81     1.57 1.00     6019     2789
muIncorrect_work_domainWeb                -0.28      0.73    -1.78     1.13 1.00     5488     3014
muIncorrect_workplace_peer_reviewfalse    -0.20      0.74    -1.67     1.26 1.00     5328     3236
muIncorrect_workplace_td_trackingfalse     0.44      0.73    -0.98     1.88 1.00     5437     3197
muNone_high_debt_versionfalse              0.30      0.61    -0.86     1.51 1.00     6084     3034
muNone_work_domainDevops                  -0.30      0.86    -2.00     1.42 1.00     6109     2828
muNone_work_domainEmbedded                -0.53      0.88    -2.23     1.20 1.00     4530     2912
muNone_work_domainFinance                 -0.31      0.77    -1.81     1.20 1.00     4634     3020
muNone_work_domainMixed                   -0.42      0.85    -2.05     1.27 1.00     6175     2912
muNone_work_domainMusic                   -0.01      0.83    -1.65     1.64 1.00     5959     3222
muNone_work_domainNone                    -0.31      0.78    -1.82     1.20 1.00     5372     2771
muNone_work_domainRetail                   0.23      0.86    -1.45     1.91 1.00     6984     2703
muNone_work_domainTelecom                  0.36      0.83    -1.31     1.98 1.00     6030     2925
muNone_work_domainWeb                      0.06      0.72    -1.35     1.51 1.00     5674     2852
muNone_workplace_peer_reviewfalse          0.99      0.73    -0.50     2.35 1.00     3836     2466
muNone_workplace_td_trackingfalse         -0.60      0.71    -2.02     0.79 1.00     5115     3024

Samples were drawn using sample(hmc). For each parameter, Bulk_ESS
and Tail_ESS are effective sample size measures, and Rhat is the potential
scale reduction factor on split chains (at convergence, Rhat = 1).
ranef(documentation3)
$session
, , muIncorrect_Intercept

                           Estimate Est.Error       Q2.5    Q97.5
6033d69a5af2c702367b3a95 -0.5603771  1.365593 -4.1833287 1.536481
6033d90a5af2c702367b3a96 -0.4378709  1.411105 -4.0232000 1.953867
6034fc165af2c702367b3a98 -0.4246564  1.325774 -3.9571182 1.797831
603500725af2c702367b3a99  1.1527434  1.630248 -0.7546270 5.406999
603f97625af2c702367b3a9d  1.2640296  1.603604 -0.6170976 5.220101
603fd5d95af2c702367b3a9e -0.4376334  1.360366 -3.9066025 1.801434
60409b7b5af2c702367b3a9f  1.1396438  1.583356 -0.7162260 5.219274
604b82b5a7718fbed181b336 -0.2858046  1.401094 -3.7536563 2.331719
6050c1bf856f36729d2e5218  1.0381249  1.550079 -0.8516789 5.075871
6050e1e7856f36729d2e5219  0.1419191  1.043177 -1.9406677 2.579764
6055fdc6856f36729d2e521b -0.3934914  1.372011 -3.8387867 2.017171
60589862856f36729d2e521f  0.3169621  1.113475 -1.7286922 2.969353
605a30a7856f36729d2e5221 -0.6565622  1.361179 -4.3442972 1.346198
605afa3a856f36729d2e5222 -0.5458650  1.363074 -4.0513580 1.581949
605c8bc6856f36729d2e5223  0.5259550  1.185150 -1.4160845 3.495365
605f3f2d856f36729d2e5224 -0.4859145  1.403190 -4.2464295 1.714097
605f46c3856f36729d2e5225 -0.6173888  1.413211 -4.3276640 1.477781
60605337856f36729d2e5226  0.2000773  1.007543 -1.7251890 2.564823
60609ae6856f36729d2e5228  0.3407595  1.099734 -1.5694170 2.974571
6061ce91856f36729d2e522e -0.6479877  1.406089 -4.3282187 1.342581
6061f106856f36729d2e5231 -0.6017436  1.359694 -4.0783642 1.402426
6068ea9f856f36729d2e523e -0.2822018  1.393687 -3.6743192 2.224814

, , muNone_Intercept

                            Estimate Est.Error       Q2.5    Q97.5
6033d69a5af2c702367b3a95 -0.16947641 1.0320587 -2.5085847 1.920971
6033d90a5af2c702367b3a96  0.76713004 1.4676916 -1.1174510 4.884850
6034fc165af2c702367b3a98  0.73337699 1.3911643 -1.0973840 4.749686
603500725af2c702367b3a99 -0.66424554 1.3994763 -4.3233720 1.360758
603f97625af2c702367b3a9d -0.70561958 1.4018144 -4.4437248 1.253827
603fd5d95af2c702367b3a9e  0.77288448 1.4467877 -0.9924838 4.579692
60409b7b5af2c702367b3a9f -0.66309446 1.4183946 -4.2448305 1.230885
604b82b5a7718fbed181b336  0.59804841 1.4144994 -1.3090165 4.335915
6050c1bf856f36729d2e5218 -0.52549501 1.3705740 -4.0863820 1.557628
6050e1e7856f36729d2e5219 -0.70485253 1.4192676 -4.5767460 1.139698
6055fdc6856f36729d2e521b  0.75187230 1.4520325 -1.1047047 4.681177
60589862856f36729d2e521f -0.86499771 1.4999719 -4.7709317 1.007139
605a30a7856f36729d2e5221  0.06077508 0.9843348 -2.0622470 2.409156
605afa3a856f36729d2e5222 -0.10507117 1.0078779 -2.4009560 1.917847
605c8bc6856f36729d2e5223  0.07513788 1.0418569 -2.0206677 2.581048
605f3f2d856f36729d2e5224  0.80558375 1.3966626 -0.8747045 4.511000
605f46c3856f36729d2e5225  1.02366435 1.4866180 -0.7480916 4.930530
60605337856f36729d2e5226 -0.74820986 1.3894169 -4.4882753 1.038718
60609ae6856f36729d2e5228  0.20695472 1.0654662 -1.8987093 2.793441
6061ce91856f36729d2e522e  0.04172705 0.9641196 -1.9494337 2.214633
6061f106856f36729d2e5231  0.97374392 1.5107063 -0.7127553 5.192776
6068ea9f856f36729d2e523e  0.59649728 1.4426868 -1.4235145 4.412081
plot(documentation3, ask = FALSE)


pp_check(documentation3, nsamples = 150, type= "bars")

Final model

All models have sampled okey but documentation3 has a bitter fit for the high debt beta parameter is significantly better than some of the simple models according to loo. wee choose documentation3 as out final model.

Trying to add covariance

documentation3_with_c <- brm(
  "documentation ~ 1 + high_debt_version + workplace_peer_review + work_domain + workplace_td_tracking + (1 | c | session)",
  prior = c(
    prior(normal(0, 0.9), class = "b"),
    prior(normal(0, 0.9), class = "Intercept"),
    prior(exponential(1), class = "sd", dpar = "muIncorrect"),
    prior(exponential(1), class = "sd", dpar = "muNone"),
    prior(lkj(2), class = "L",)
  ),
  family = categorical(),
  data = as.data.frame(d.both_completed),
  file = "fits/documentation3_with_c",
  file_refit = "on_change"
)
Specifying global priors for regression coefficients in categorical models is deprecated. Please specify priors separately for each response category.Specifying global priors for regression coefficients in categorical models is deprecated. Please specify priors separately for each response category.Compiling Stan program...

-

\

|

/

-

\

|

/

-

\

|

/

-

\

|

/

-

\

|

/

-

\

|

/

-

\

|

/

-

\

|

/

-

\

|

/

-

\

|

/

-

\

|

/

-

\

|

/

-

\

|

/

-

\

|

/

-

\

|

/

-

\

|

/

-

\

|

/

-

\

|

/

-

\

|

/

-

\

|

/

-

\

|

/

-

\

|

/

-

\

|

/

-

\

|

/

-

\

|

/

-

\

|

/

-

\

|

/

-

\

|

/

-

\

|

/

-

\

|

/

-

\

|

/

-

\

|

/

-

\

|

/

-

 
Start sampling
Running MCMC with 4 parallel chains...

Chain 1 Iteration:    1 / 2000 [  0%]  (Warmup) 
Chain 1 Iteration:  100 / 2000 [  5%]  (Warmup) 
Chain 1 Iteration:  200 / 2000 [ 10%]  (Warmup) 
Chain 2 Iteration:    1 / 2000 [  0%]  (Warmup) 
Chain 2 Iteration:  100 / 2000 [  5%]  (Warmup) 
Chain 2 Iteration:  200 / 2000 [ 10%]  (Warmup) 
Chain 3 Iteration:    1 / 2000 [  0%]  (Warmup) 
Chain 3 Iteration:  100 / 2000 [  5%]  (Warmup) 
Chain 3 Iteration:  200 / 2000 [ 10%]  (Warmup) 
Chain 4 Iteration:    1 / 2000 [  0%]  (Warmup) 
Chain 4 Iteration:  100 / 2000 [  5%]  (Warmup) 
Chain 4 Iteration:  200 / 2000 [ 10%]  (Warmup) 
Chain 4 Iteration:  300 / 2000 [ 15%]  (Warmup) 
Chain 1 Iteration:  300 / 2000 [ 15%]  (Warmup) 
Chain 1 Iteration:  400 / 2000 [ 20%]  (Warmup) 
Chain 1 Iteration:  500 / 2000 [ 25%]  (Warmup) 
Chain 1 Iteration:  600 / 2000 [ 30%]  (Warmup) 
Chain 1 Iteration:  700 / 2000 [ 35%]  (Warmup) 
Chain 2 Iteration:  300 / 2000 [ 15%]  (Warmup) 
Chain 2 Iteration:  400 / 2000 [ 20%]  (Warmup) 
Chain 2 Iteration:  500 / 2000 [ 25%]  (Warmup) 
Chain 3 Iteration:  300 / 2000 [ 15%]  (Warmup) 
Chain 3 Iteration:  400 / 2000 [ 20%]  (Warmup) 
Chain 4 Iteration:  400 / 2000 [ 20%]  (Warmup) 
Chain 4 Iteration:  500 / 2000 [ 25%]  (Warmup) 
Chain 1 Iteration:  800 / 2000 [ 40%]  (Warmup) 
Chain 2 Iteration:  600 / 2000 [ 30%]  (Warmup) 
Chain 3 Iteration:  500 / 2000 [ 25%]  (Warmup) 
Chain 4 Iteration:  600 / 2000 [ 30%]  (Warmup) 
Chain 1 Iteration:  900 / 2000 [ 45%]  (Warmup) 
Chain 1 Iteration: 1000 / 2000 [ 50%]  (Warmup) 
Chain 2 Iteration:  700 / 2000 [ 35%]  (Warmup) 
Chain 2 Iteration:  800 / 2000 [ 40%]  (Warmup) 
Chain 3 Iteration:  600 / 2000 [ 30%]  (Warmup) 
Chain 4 Iteration:  700 / 2000 [ 35%]  (Warmup) 
Chain 4 Iteration:  800 / 2000 [ 40%]  (Warmup) 
Chain 1 Iteration: 1001 / 2000 [ 50%]  (Sampling) 
Chain 2 Iteration:  900 / 2000 [ 45%]  (Warmup) 
Chain 3 Iteration:  700 / 2000 [ 35%]  (Warmup) 
Chain 4 Iteration:  900 / 2000 [ 45%]  (Warmup) 
Chain 1 Iteration: 1100 / 2000 [ 55%]  (Sampling) 
Chain 2 Iteration: 1000 / 2000 [ 50%]  (Warmup) 
Chain 2 Iteration: 1001 / 2000 [ 50%]  (Sampling) 
Chain 3 Iteration:  800 / 2000 [ 40%]  (Warmup) 
Chain 3 Iteration:  900 / 2000 [ 45%]  (Warmup) 
Chain 4 Iteration: 1000 / 2000 [ 50%]  (Warmup) 
Chain 4 Iteration: 1001 / 2000 [ 50%]  (Sampling) 
Chain 1 Iteration: 1200 / 2000 [ 60%]  (Sampling) 
Chain 3 Iteration: 1000 / 2000 [ 50%]  (Warmup) 
Chain 3 Iteration: 1001 / 2000 [ 50%]  (Sampling) 
Chain 4 Iteration: 1100 / 2000 [ 55%]  (Sampling) 
Chain 1 Iteration: 1300 / 2000 [ 65%]  (Sampling) 
Chain 2 Iteration: 1100 / 2000 [ 55%]  (Sampling) 
Chain 3 Iteration: 1100 / 2000 [ 55%]  (Sampling) 
Chain 4 Iteration: 1200 / 2000 [ 60%]  (Sampling) 
Chain 4 Iteration: 1300 / 2000 [ 65%]  (Sampling) 
Chain 1 Iteration: 1400 / 2000 [ 70%]  (Sampling) 
Chain 2 Iteration: 1200 / 2000 [ 60%]  (Sampling) 
Chain 3 Iteration: 1200 / 2000 [ 60%]  (Sampling) 
Chain 1 Iteration: 1500 / 2000 [ 75%]  (Sampling) 
Chain 2 Iteration: 1300 / 2000 [ 65%]  (Sampling) 
Chain 3 Iteration: 1300 / 2000 [ 65%]  (Sampling) 
Chain 4 Iteration: 1400 / 2000 [ 70%]  (Sampling) 
Chain 1 Iteration: 1600 / 2000 [ 80%]  (Sampling) 
Chain 3 Iteration: 1400 / 2000 [ 70%]  (Sampling) 
Chain 4 Iteration: 1500 / 2000 [ 75%]  (Sampling) 
Chain 1 Iteration: 1700 / 2000 [ 85%]  (Sampling) 
Chain 2 Iteration: 1400 / 2000 [ 70%]  (Sampling) 
Chain 3 Iteration: 1500 / 2000 [ 75%]  (Sampling) 
Chain 4 Iteration: 1600 / 2000 [ 80%]  (Sampling) 
Chain 1 Iteration: 1800 / 2000 [ 90%]  (Sampling) 
Chain 2 Iteration: 1500 / 2000 [ 75%]  (Sampling) 
Chain 3 Iteration: 1600 / 2000 [ 80%]  (Sampling) 
Chain 4 Iteration: 1700 / 2000 [ 85%]  (Sampling) 
Chain 1 Iteration: 1900 / 2000 [ 95%]  (Sampling) 
Chain 3 Iteration: 1700 / 2000 [ 85%]  (Sampling) 
Chain 4 Iteration: 1800 / 2000 [ 90%]  (Sampling) 
Chain 2 Iteration: 1600 / 2000 [ 80%]  (Sampling) 
Chain 4 Iteration: 1900 / 2000 [ 95%]  (Sampling) 
Chain 1 Iteration: 2000 / 2000 [100%]  (Sampling) 
Chain 3 Iteration: 1800 / 2000 [ 90%]  (Sampling) 
Chain 1 finished in 2.2 seconds.
Chain 2 Iteration: 1700 / 2000 [ 85%]  (Sampling) 
Chain 3 Iteration: 1900 / 2000 [ 95%]  (Sampling) 
Chain 4 Iteration: 2000 / 2000 [100%]  (Sampling) 
Chain 4 finished in 2.2 seconds.
Chain 2 Iteration: 1800 / 2000 [ 90%]  (Sampling) 
Chain 3 Iteration: 2000 / 2000 [100%]  (Sampling) 
Chain 3 finished in 2.4 seconds.
Chain 2 Iteration: 1900 / 2000 [ 95%]  (Sampling) 
Chain 2 Iteration: 2000 / 2000 [100%]  (Sampling) 
Chain 2 finished in 2.7 seconds.

All 4 chains finished successfully.
Mean chain execution time: 2.4 seconds.
Total execution time: 2.9 seconds.
Registered S3 method overwritten by 'data.table':
  method           from
  print.data.table     
summary(documentation3_with_c)
 Family: categorical 
  Links: muIncorrect = logit; muNone = logit 
Formula: documentation ~ 1 + high_debt_version + workplace_peer_review + work_domain + workplace_td_tracking + (1 | c | session) 
   Data: as.data.frame(d.both_completed) (Number of observations: 44) 
Samples: 4 chains, each with iter = 2000; warmup = 1000; thin = 1;
         total post-warmup samples = 4000

Group-Level Effects: 
~session (Number of levels: 22) 
                                            Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS Tail_ESS
sd(muIncorrect_Intercept)                       1.22      0.97     0.04     3.59 1.00     1141     1987
sd(muNone_Intercept)                            1.21      0.95     0.04     3.60 1.00     1238     2277
cor(muIncorrect_Intercept,muNone_Intercept)    -0.17      0.43    -0.86     0.72 1.00     2419     2619

Population-Level Effects: 
                                       Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS Tail_ESS
muIncorrect_Intercept                     -0.18      0.89    -1.95     1.54 1.00     5323     3288
muNone_Intercept                           0.85      0.80    -0.76     2.38 1.00     5920     3224
muIncorrect_high_debt_versionfalse        -0.09      0.63    -1.34     1.13 1.00     8422     2880
muIncorrect_workplace_peer_reviewfalse    -0.15      0.75    -1.64     1.34 1.00     6767     2502
muIncorrect_work_domainDevops              0.51      0.86    -1.16     2.20 1.00     6404     2953
muIncorrect_work_domainEmbedded            0.14      0.83    -1.51     1.76 1.00     6522     2842
muIncorrect_work_domainFinance            -0.51      0.83    -2.15     1.14 1.00     7026     2872
muIncorrect_work_domainMixed               0.04      0.83    -1.63     1.66 1.00     8060     2902
muIncorrect_work_domainMusic               0.26      0.86    -1.45     2.00 1.00     7085     2696
muIncorrect_work_domainNone                0.89      0.78    -0.65     2.37 1.00     5873     3152
muIncorrect_work_domainRetail             -0.12      0.90    -1.84     1.62 1.00     7808     3043
muIncorrect_work_domainTelecom            -0.17      0.86    -1.83     1.51 1.00     7942     3004
muIncorrect_work_domainWeb                -0.27      0.74    -1.78     1.21 1.00     6352     3191
muIncorrect_workplace_td_trackingfalse     0.44      0.76    -1.07     1.93 1.00     6027     2948
muNone_high_debt_versionfalse              0.32      0.60    -0.84     1.51 1.00     6829     2873
muNone_workplace_peer_reviewfalse          1.00      0.72    -0.39     2.35 1.00     5148     3092
muNone_work_domainDevops                  -0.27      0.86    -1.93     1.43 1.00     6542     2769
muNone_work_domainEmbedded                -0.50      0.84    -2.15     1.16 1.00     6381     3161
muNone_work_domainFinance                 -0.30      0.79    -1.84     1.30 1.00     6143     3353
muNone_work_domainMixed                   -0.41      0.85    -2.08     1.25 1.00     6550     2801
muNone_work_domainMusic                   -0.01      0.84    -1.63     1.59 1.00     7421     3059
muNone_work_domainNone                    -0.27      0.79    -1.78     1.25 1.00     6824     3123
muNone_work_domainRetail                   0.22      0.87    -1.46     1.93 1.00     7755     2985
muNone_work_domainTelecom                  0.33      0.86    -1.32     2.06 1.00     6296     2728
muNone_work_domainWeb                      0.05      0.73    -1.38     1.48 1.00     7448     2919
muNone_workplace_td_trackingfalse         -0.57      0.71    -1.99     0.84 1.00     6388     3297

Samples were drawn using sample(hmc). For each parameter, Bulk_ESS
and Tail_ESS are effective sample size measures, and Rhat is the potential
scale reduction factor on split chains (at convergence, Rhat = 1).
ranef(documentation3_with_c)
$session
, , muIncorrect_Intercept

                           Estimate Est.Error       Q2.5    Q97.5
6033d69a5af2c702367b3a95 -0.4740170  1.349549 -3.8904217 1.622149
6033d90a5af2c702367b3a96 -0.5486771  1.422207 -4.2951387 1.788122
6034fc165af2c702367b3a98 -0.5364618  1.420365 -4.3462845 1.694847
603500725af2c702367b3a99  1.1888933  1.665348 -0.6752396 5.454351
603f97625af2c702367b3a9d  1.2871637  1.554135 -0.5700177 5.168141
603fd5d95af2c702367b3a9e -0.5383334  1.395016 -4.1240832 1.656860
60409b7b5af2c702367b3a9f  1.1492655  1.557404 -0.6702289 5.124011
604b82b5a7718fbed181b336 -0.3947439  1.428949 -3.9845432 2.057823
6050c1bf856f36729d2e5218  1.0869348  1.574547 -0.6869054 5.109583
6050e1e7856f36729d2e5219  0.2284617  1.050561 -1.8104610 2.668000
6055fdc6856f36729d2e521b -0.5149120  1.391413 -4.0554980 1.654794
60589862856f36729d2e521f  0.4047294  1.119904 -1.6248867 3.103528
605a30a7856f36729d2e5221 -0.6305685  1.385323 -4.2688537 1.414821
605afa3a856f36729d2e5222 -0.5260204  1.350655 -4.0110530 1.556350
605c8bc6856f36729d2e5223  0.4871209  1.183538 -1.4799500 3.482942
605f3f2d856f36729d2e5224 -0.5572856  1.383517 -4.0131750 1.559544
605f46c3856f36729d2e5225 -0.7259060  1.464672 -4.3730900 1.305701
60605337856f36729d2e5226  0.3123440  1.023106 -1.5360945 2.745020
60609ae6856f36729d2e5228  0.2911768  1.105874 -1.7371525 2.962782
6061ce91856f36729d2e522e -0.6295638  1.354371 -4.1986288 1.284592
6061f106856f36729d2e5231 -0.7161355  1.403589 -4.4195718 1.209440
6068ea9f856f36729d2e523e -0.4117994  1.434870 -4.0406335 2.093152

, , muNone_Intercept

                             Estimate Est.Error       Q2.5     Q97.5
6033d69a5af2c702367b3a95 -0.095175704 1.0282401 -2.4283315 2.0582043
6033d90a5af2c702367b3a96  0.869406506 1.4682907 -1.0033360 4.5763228
6034fc165af2c702367b3a98  0.826466220 1.5007878 -1.1077500 4.6382268
603500725af2c702367b3a99 -0.830604253 1.4675248 -4.6771415 1.1719215
603f97625af2c702367b3a9d -0.866577467 1.4771711 -4.8287575 1.1342350
603fd5d95af2c702367b3a9e  0.864852450 1.5246855 -1.0572250 4.8986993
60409b7b5af2c702367b3a9f -0.831289932 1.4408650 -4.5627458 1.0589548
604b82b5a7718fbed181b336  0.671833314 1.4217659 -1.3227875 4.3870320
6050c1bf856f36729d2e5218 -0.687363331 1.4440616 -4.3477467 1.4542002
6050e1e7856f36729d2e5219 -0.717601888 1.4426154 -4.5868553 1.2550483
6055fdc6856f36729d2e521b  0.876459045 1.4810539 -1.0061095 4.7831158
60589862856f36729d2e521f -0.860613797 1.4319841 -4.3080075 1.0334343
605a30a7856f36729d2e5221  0.107802137 0.9669728 -1.8747055 2.2739908
605afa3a856f36729d2e5222 -0.023811007 1.0210658 -2.2640835 2.2188023
605c8bc6856f36729d2e5223  0.005140308 1.0488520 -2.2531653 2.3370798
605f3f2d856f36729d2e5224  0.902602636 1.4443933 -0.8216154 4.7493038
605f46c3856f36729d2e5225  1.086796364 1.4870779 -0.6899161 4.8028003
60605337856f36729d2e5226 -0.795304877 1.3837966 -4.4117265 0.9448756
60609ae6856f36729d2e5228  0.129812004 1.0106389 -1.8727820 2.4984315
6061ce91856f36729d2e522e  0.109028807 0.9949454 -1.9458302 2.3347675
6061f106856f36729d2e5231  1.038764660 1.5270026 -0.7500039 4.9471303
6068ea9f856f36729d2e523e  0.700633994 1.4825041 -1.3700782 4.6034325
plot(documentation3_with_c, ask = FALSE)


pp_check(documentation3_with_c, nsamples = 150, type= "bars")

loo(
  documentation3,
  documentation3_with_c
)
Found 4 observations with a pareto_k > 0.7 in model 'documentation3'. It is recommended to set 'moment_match = TRUE' in order to perform moment matching for problematic observations. Found 3 observations with a pareto_k > 0.7 in model 'documentation3_with_c'. It is recommended to set 'moment_match = TRUE' in order to perform moment matching for problematic observations. 
Output of model 'documentation3':

Computed from 4000 by 44 log-likelihood matrix
------
Monte Carlo SE of elpd_loo is NA.

Pareto k diagnostic values:
                         Count Pct.    Min. n_eff
(-Inf, 0.5]   (good)     29    65.9%   1023      
 (0.5, 0.7]   (ok)       11    25.0%   675       
   (0.7, 1]   (bad)       4     9.1%   245       
   (1, Inf)   (very bad)  0     0.0%   <NA>      
See help('pareto-k-diagnostic') for details.

Output of model 'documentation3_with_c':

Computed from 4000 by 44 log-likelihood matrix
------
Monte Carlo SE of elpd_loo is NA.

Pareto k diagnostic values:
                         Count Pct.    Min. n_eff
(-Inf, 0.5]   (good)     28    63.6%   835       
 (0.5, 0.7]   (ok)       13    29.5%   458       
   (0.7, 1]   (bad)       3     6.8%   367       
   (1, Inf)   (very bad)  0     0.0%   <NA>      
See help('pareto-k-diagnostic') for details.

Model comparisons:
                      elpd_diff se_diff
documentation3_with_c  0.0       0.0   
documentation3        -0.3       0.3   

Adding the covariance parameter did not significantly improve the model and we will therefore continue to use the simpler model documentation3.

Model with with incomplete data points

Some participants did only complete one scenario. Those has been excluded from the initial dataset to improve sampling of the models. We do however want to use all data we can and will therefore try to fit the model with the complete dataset.

documentation3.all <- brm(
  "documentation ~ 1 + high_debt_version + workplace_peer_review + work_domain + workplace_td_tracking + (1 | session)",
  prior = c(
    prior(normal(0, 0.9), class = "b"),
    prior(normal(0, 0.9), class = "Intercept"),
    prior(exponential(1), class = "sd", dpar = "muIncorrect"),
    prior(exponential(1), class = "sd", dpar = "muNone")
  ),
  family = categorical(),
  data = as.data.frame(d.completed),
  file = "fits/documentation3.all",
  file_refit = "on_change"
)
Specifying global priors for regression coefficients in categorical models is deprecated. Please specify priors separately for each response category.Specifying global priors for regression coefficients in categorical models is deprecated. Please specify priors separately for each response category.
summary(documentation3.all)
 Family: categorical 
  Links: muIncorrect = logit; muNone = logit 
Formula: documentation ~ 1 + high_debt_version + workplace_peer_review + work_domain + workplace_td_tracking + (1 | session) 
   Data: as.data.frame(d.completed) (Number of observations: 49) 
Samples: 4 chains, each with iter = 2000; warmup = 1000; thin = 1;
         total post-warmup samples = 4000

Group-Level Effects: 
~session (Number of levels: 27) 
                          Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS Tail_ESS
sd(muIncorrect_Intercept)     1.06      0.88     0.03     3.24 1.00      998     1808
sd(muNone_Intercept)          0.99      0.86     0.03     3.25 1.01      855     1697

Population-Level Effects: 
                                       Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS Tail_ESS
muIncorrect_Intercept                      0.00      0.88    -1.78     1.65 1.00     3786     2586
muNone_Intercept                           0.77      0.81    -0.82     2.33 1.00     3981     2632
muIncorrect_high_debt_versionfalse        -0.35      0.64    -1.63     0.91 1.00     3879     2558
muIncorrect_workplace_peer_reviewfalse    -0.06      0.71    -1.45     1.33 1.00     4096     2720
muIncorrect_work_domainAutomotive         -0.49      0.78    -2.03     1.02 1.00     4267     2655
muIncorrect_work_domainDevops              0.51      0.85    -1.16     2.15 1.00     5133     2988
muIncorrect_work_domainEMCommerce          0.30      0.88    -1.43     2.04 1.00     5377     2845
muIncorrect_work_domainEmbedded            0.17      0.85    -1.45     1.83 1.00     5257     2458
muIncorrect_work_domainFinance            -0.54      0.82    -2.15     1.09 1.00     3847     2714
muIncorrect_work_domainMixed               0.02      0.84    -1.62     1.63 1.00     5104     2788
muIncorrect_work_domainMusic               0.28      0.84    -1.40     1.91 1.00     5266     2831
muIncorrect_work_domainNone                0.86      0.73    -0.56     2.29 1.00     3809     2573
muIncorrect_work_domainRetail             -0.16      0.86    -1.88     1.52 1.00     5015     2441
muIncorrect_work_domainTelecom            -0.22      0.85    -1.88     1.44 1.00     5254     2855
muIncorrect_work_domainWeb                -0.44      0.73    -1.86     0.95 1.00     5044     3030
muIncorrect_workplace_td_trackingfalse     0.58      0.75    -0.89     2.01 1.00     4111     2895
muNone_high_debt_versionfalse              0.37      0.60    -0.81     1.57 1.00     3991     3132
muNone_workplace_peer_reviewfalse          1.00      0.69    -0.39     2.31 1.00     3357     2924
muNone_work_domainAutomotive               0.76      0.72    -0.69     2.13 1.00     4474     2700
muNone_work_domainDevops                  -0.27      0.86    -1.97     1.38 1.00     4875     2761
muNone_work_domainEMCommerce              -0.15      0.86    -1.87     1.54 1.00     5608     3018
muNone_work_domainEmbedded                -0.53      0.86    -2.22     1.13 1.00     4529     3115
muNone_work_domainFinance                 -0.24      0.79    -1.79     1.30 1.00     4305     2587
muNone_work_domainMixed                   -0.35      0.87    -2.10     1.33 1.00     4934     3109
muNone_work_domainMusic                   -0.00      0.86    -1.70     1.72 1.00     4805     2833
muNone_work_domainNone                    -0.05      0.73    -1.43     1.38 1.00     4750     2996
muNone_work_domainRetail                   0.25      0.85    -1.37     1.87 1.00     5266     2843
muNone_work_domainTelecom                  0.40      0.87    -1.27     2.11 1.00     5359     3074
muNone_work_domainWeb                      0.40      0.69    -0.99     1.79 1.00     4645     3143
muNone_workplace_td_trackingfalse         -0.84      0.71    -2.24     0.57 1.00     3667     2220

Samples were drawn using sample(hmc). For each parameter, Bulk_ESS
and Tail_ESS are effective sample size measures, and Rhat is the potential
scale reduction factor on split chains (at convergence, Rhat = 1).
ranef(documentation3.all)
$session
, , muIncorrect_Intercept

                            Estimate Est.Error       Q2.5    Q97.5
6033c6fc5af2c702367b3a93  0.56668439 1.3900710 -1.3564902 4.358797
6033d69a5af2c702367b3a95 -0.51816620 1.2588619 -3.9174198 1.362781
6033d90a5af2c702367b3a96 -0.30818290 1.2365050 -3.3053398 1.836268
6034fc165af2c702367b3a98 -0.35746056 1.2515824 -3.5075447 1.687832
603500725af2c702367b3a99  0.91078960 1.4807329 -0.8130972 5.051990
603f84f15af2c702367b3a9b -0.46378477 1.3007111 -3.8238760 1.598761
603f97625af2c702367b3a9d  1.09826959 1.4915761 -0.5676147 5.028738
603fd5d95af2c702367b3a9e -0.32641539 1.2325152 -3.4746550 1.755247
60409b7b5af2c702367b3a9f  0.90267517 1.4050692 -0.7899109 4.615886
604b82b5a7718fbed181b336 -0.21797633 1.2297951 -3.2245887 1.984291
604f1239a7718fbed181b33f  0.57575704 1.3356545 -1.2706010 4.265109
6050c1bf856f36729d2e5218  0.84327015 1.4497551 -0.9712524 4.842020
6050e1e7856f36729d2e5219  0.09232487 1.0199872 -1.8919607 2.392772
6055fdc6856f36729d2e521b -0.35640012 1.2716045 -3.5745012 1.728544
60579f2a856f36729d2e521e -0.29381870 1.2108395 -3.2560522 1.912461
60589862856f36729d2e521f  0.26026107 1.0077301 -1.6189578 2.779680
605a30a7856f36729d2e5221 -0.58023210 1.2833446 -4.0179022 1.251975
605afa3a856f36729d2e5222 -0.47011977 1.3125490 -3.9474637 1.515763
605c8bc6856f36729d2e5223  0.42659282 1.0888754 -1.2925652 3.231538
605f3f2d856f36729d2e5224 -0.29479752 1.2642510 -3.3185883 1.942462
605f46c3856f36729d2e5225 -0.44612047 1.2838081 -3.8377990 1.523013
60605337856f36729d2e5226  0.34015025 1.0587691 -1.5499285 3.050282
60609ae6856f36729d2e5228  0.16906903 0.9865542 -1.7369890 2.639110
6061ce91856f36729d2e522e -0.56594985 1.2566168 -3.9529910 1.183943
6061f106856f36729d2e5231 -0.62303299 1.2602445 -3.9279680 1.081355
60672faa856f36729d2e523c -0.34915974 1.2340610 -3.4919848 1.700463
6068ea9f856f36729d2e523e -0.29085361 1.3114368 -3.6383952 1.900159

, , muNone_Intercept

                            Estimate Est.Error       Q2.5     Q97.5
6033c6fc5af2c702367b3a93 -0.31975767 1.2189100 -3.3697577 1.6906123
6033d69a5af2c702367b3a95 -0.06278226 0.9273268 -2.1735618 1.9110213
6033d90a5af2c702367b3a96  0.58113279 1.3059730 -1.0898352 4.0925765
6034fc165af2c702367b3a98  0.62740112 1.2820138 -1.0774657 4.0800975
603500725af2c702367b3a99 -0.50791441 1.1806103 -3.6122665 1.1815005
603f84f15af2c702367b3a9b -0.42847632 1.2603699 -3.8646552 1.4205428
603f97625af2c702367b3a9d -0.59629911 1.2798042 -4.1252695 1.0923448
603fd5d95af2c702367b3a9e  0.55655334 1.2269826 -1.1445050 3.9102400
60409b7b5af2c702367b3a9f -0.52608135 1.2255266 -3.8040947 1.2553955
604b82b5a7718fbed181b336  0.47987881 1.2396653 -1.2894017 3.7864998
604f1239a7718fbed181b33f -0.29773930 1.2350288 -3.4176547 1.8673838
6050c1bf856f36729d2e5218 -0.41232613 1.2413335 -3.7342760 1.4039380
6050e1e7856f36729d2e5219 -0.53521184 1.2406922 -3.6471362 1.2327300
6055fdc6856f36729d2e521b  0.63073666 1.3052796 -1.1016473 4.1732833
60579f2a856f36729d2e521e  0.50515355 1.2151732 -1.2383735 3.7816730
60589862856f36729d2e521f -0.70828053 1.3667908 -4.4140030 0.9217694
605a30a7856f36729d2e5221  0.02511206 0.9109578 -1.9496705 2.0509803
605afa3a856f36729d2e5222 -0.06936711 0.9714361 -2.2305623 1.9991320
605c8bc6856f36729d2e5223  0.06387040 0.9649576 -1.8657480 2.3023402
605f3f2d856f36729d2e5224  0.52634041 1.2488189 -1.2207750 3.7981818
605f46c3856f36729d2e5225  0.72517113 1.2475205 -0.8546529 4.0628638
60605337856f36729d2e5226 -0.74071093 1.2736229 -4.1198245 0.7786216
60609ae6856f36729d2e5228  0.20337996 0.9805332 -1.6567338 2.5486325
6061ce91856f36729d2e522e  0.05958744 0.8940686 -1.8627612 2.0501413
6061f106856f36729d2e5231  0.86014412 1.4043814 -0.7155665 4.7656840
60672faa856f36729d2e523c  0.53896524 1.2493469 -1.2020765 4.0047203
6068ea9f856f36729d2e523e  0.48951876 1.2275134 -1.3053530 3.7424770
plot(documentation3.all, ask = FALSE)


pp_check(documentation3_with_c, nsamples = 150, type= "bars")

Training the model on all data points reduces the uncertainty and did not result in any sampling problems. We will proceed with the model fitted to all the data.

Interpreting the model

Extract posterior samples:


post <- posterior_predict(documentation3.all, newdata = data.frame(
  high_debt_version = c("false", "true"),
  session = NA, 
  workplace_peer_review = NA, 
  work_domain = NA, 
  workplace_td_tracking = NA
))
post.low <-  post[,1]
post.high <- post[,2]
summary(post)
       V1              V2       
 Min.   :1.000   Min.   :1.000  
 1st Qu.:2.000   1st Qu.:2.000  
 Median :3.000   Median :2.000  
 Mean   :2.383   Mean   :2.268  
 3rd Qu.:3.000   3rd Qu.:3.000  
 Max.   :3.000   Max.   :3.000  
LS0tCnRpdGxlOiAiRG9jdW1lbnRhdGlvbiIKb3V0cHV0OiBodG1sX25vdGVib29rCi0tLQoKIyMgTG9va2luZyBhdCB0aGUgZGF0YQpXZSBwbG90IHRoZSBkYXRhIGFuZCBjYW4gc2VlIHRoYXQgdGhlcmUgaXMgbm8gb2J2aW91cyBsYXJnZSBkaWZmZXJlbmNlIGJldHdlZW4gdGhlIGRlYnQgdmVyc2lvbnMuCgpgYGB7cn0KZC5ib3RoX2NvbXBsZXRlZCAlPiUKICBnZ3Bsb3QoYWVzKGhpZ2hfZGVidF92ZXJzaW9uLCBmaWxsID0gZG9jdW1lbnRhdGlvbikpICsKICBnZW9tX2Jhcihwb3NpdGlvbiA9ICJmaWxsIikgKwogIHNjYWxlX3lfcmV2ZXJzZSgpICsKICBzY2FsZV9maWxsX21hbnVhbCgibGVnZW5kIiwgdmFsdWVzID0gYygiQ29ycmVjdCIgPSAiZ3JlZW4iLCAiTm9uZSIgPSAieWVsbG93IiwgIkluY29ycmVjdCIgPSAicmVkIiksIGd1aWRlID0gZ3VpZGVfbGVnZW5kKHJldmVyc2UgPSBUUlVFKSkKYGBgCiMjIEluaXRpYWwgbW9kZWwKVGhlIHR5cGUgb2YgdGhlIG91dGNvbWUgaXMgYWRqYWNlbnQgY2F0ZWdvcmllcyBhbmQgaXMgbW9kZWxlZCBhcyBhIGN1bXVsYXRpdmUgZmFtaWx5LiBUT0RPIEZJWCEKCldlIGluY2x1ZGUgYGhpZ2hfZGVidF92ZXJpc29uYCBhcyB3ZWxsIGFzIGEgdmFyeWluZyBpbnRlcmNlcHQgZm9yIGVhY2ggaW5kaXZpZHVhbCBpbiBvdXIgaW5pdGlhbCBtb2RlbC4KCiMjIyBTZWxlY3RpbmcgUHJpb3JzCldlIGl0ZXJhdGUgb3ZlciB0aGUgbW9kZWwgdW50aWwgd2UgaGF2ZSBzYW5lIHByaW9ycwoKYGBge3J9Cgpkb2N1bWVudGF0aW9uMC53aXRoIDwtIGV4dGVuZGFibGVfbW9kZWwoCiAgYmFzZV9uYW1lID0gImRvY3VtZW50YXRpb24wIiwKICBiYXNlX2Zvcm11bGEgPSAiZG9jdW1lbnRhdGlvbiB+IDEgKyBoaWdoX2RlYnRfdmVyc2lvbiArICgxIHwgc2Vzc2lvbikiLAogIGJhc2VfcHJpb3JzID0gYygKICAgIHByaW9yKG5vcm1hbCgwLCAwLjkpLCBjbGFzcyA9ICJiIiksCiAgICBwcmlvcihub3JtYWwoMCwgMC45KSwgY2xhc3MgPSAiSW50ZXJjZXB0IiksCiAgICBwcmlvcihleHBvbmVudGlhbCgxKSwgY2xhc3MgPSAic2QiLCBkcGFyID0gIm11SW5jb3JyZWN0IiksCiAgICBwcmlvcihleHBvbmVudGlhbCgxKSwgY2xhc3MgPSAic2QiLCBkcGFyID0gIm11Tm9uZSIpCiAgKSwKICBmYW1pbHkgPSBjYXRlZ29yaWNhbCgpLAogIGRhdGEgPSBkLmJvdGhfY29tcGxldGVkLAopCgojIERlZmF1bHQgcHJpb3JzOgpwcmlvcl9zdW1tYXJ5KGRvY3VtZW50YXRpb24wLndpdGgob25seV9wcmlvcnM9IFRSVUUpKQoKIyBPdXIgcHJpb3JzOgpwcmlvcl9zdW1tYXJ5KGRvY3VtZW50YXRpb24wLndpdGgoc2FtcGxlX3ByaW9yID0gIm9ubHkiKSkKCiMgUHJpb3IgcHJlZGljdGl2ZSBjaGVjawpwcF9jaGVjayhkb2N1bWVudGF0aW9uMC53aXRoKHNhbXBsZV9wcmlvciA9ICJvbmx5IiksIG5zYW1wbGVzID0gMjAwLCB0eXBlID0gImJhcnMiKQpgYGAKCgojIyMgTW9kZWwgZml0CldlIGNoZWNrIHRoZSBwb3N0ZXJpb3IgZGlzdHJpYnV0aW9uIGFuZCBjYW4gc2VlIHRoYXQgdGhlIG1vZGVsIHNlZW1zIHRvIGhhdmUgYmVlbiBhYmxlIHRvIGZpdCB0aGUgZGF0YSB3ZWxsCmBgYHtyfQojIFBvc3RlcmlvciBwcmVkaWN0aXZlIGNoZWNrCnBwX2NoZWNrKGRvY3VtZW50YXRpb24wLndpdGgoKSwgbnNhbXBsZXMgPSAxMDAsIHR5cGUgPSAiYmFycyIpCgpzdW1tYXJ5KGRvY3VtZW50YXRpb24wLndpdGgoKSkKYGBgCiMjIE1vZGVsIGV4dGVuc3Rpb25zCgpXZSB1c2UgbG9vIHRvIGNoZWNrIHNvbWUgcG9zc2libGUgZXh0ZW5zaW9ucyBvbiB0aGUgbW9kZWwuCgpgYGB7cn0KCmVkbHZsX3ByaW9yIDwtIGMoCiAgcHJpb3IoZGlyaWNobGV0KDIpLCBjbGFzcyA9ICJzaW1vIiwgY29lZiA9ICJtb2VkdWNhdGlvbl9sZXZlbDEiLCBkcGFyID0gIm11SW5jb3JyZWN0IiksCiAgcHJpb3IoZGlyaWNobGV0KDIpLCBjbGFzcyA9ICJzaW1vIiwgY29lZiA9ICJtb2VkdWNhdGlvbl9sZXZlbDEiLCBkcGFyID0gIm11Tm9uZSIpCikKCmxvbygKICBkb2N1bWVudGF0aW9uMC53aXRoKCksCiAgZG9jdW1lbnRhdGlvbjAud2l0aCgid29ya19kb21haW4iKSwKICBkb2N1bWVudGF0aW9uMC53aXRoKCJ3b3JrX2V4cGVyaWVuY2VfcHJvZ3JhbW1pbmcucyIpLAogIGRvY3VtZW50YXRpb24wLndpdGgoIndvcmtfZXhwZXJpZW5jZV9qYXZhLnMiKSwKICBkb2N1bWVudGF0aW9uMC53aXRoKCJlZHVjYXRpb25fZmllbGQiKSwKICBkb2N1bWVudGF0aW9uMC53aXRoKCJtbyhlZHVjYXRpb25fbGV2ZWwpIiwgZWRsdmxfcHJpb3IpLAogIGRvY3VtZW50YXRpb24wLndpdGgoIndvcmtwbGFjZV9wZWVyX3JldmlldyIpLAogIGRvY3VtZW50YXRpb24wLndpdGgoIndvcmtwbGFjZV90ZF90cmFja2luZyIpLAogIGRvY3VtZW50YXRpb24wLndpdGgoIndvcmtwbGFjZV9wYWlyX3Byb2dyYW1taW5nIiksCiAgZG9jdW1lbnRhdGlvbjAud2l0aCgid29ya3BsYWNlX2NvZGluZ19zdGFuZGFyZHMiKSwKICBkb2N1bWVudGF0aW9uMC53aXRoKCJzY2VuYXJpbyIpLAogIGRvY3VtZW50YXRpb24wLndpdGgoImdyb3VwIiksCiAgZG9jdW1lbnRhdGlvbjAud2l0aCgib3JkZXIiKQopCmBgYAoKV2UgcGljayBvdXQgc29tZSBpbnRlcmVzdGluZyB2YXJpYWJsZXMgYW5kIHRyeSBjb21iaW5pbmcgdGhlbQoKYGBge3J9CmxvbygKICBkb2N1bWVudGF0aW9uMC53aXRoKCksCiAgZG9jdW1lbnRhdGlvbjAud2l0aCgid29ya19kb21haW4iKSwKICBkb2N1bWVudGF0aW9uMC53aXRoKCJ3b3JrcGxhY2VfcGVlcl9yZXZpZXciKSwKICBkb2N1bWVudGF0aW9uMC53aXRoKCJ3b3JrcGxhY2VfdGRfdHJhY2tpbmciKSwKICAKICBkb2N1bWVudGF0aW9uMC53aXRoKGMoIndvcmtfZG9tYWluIiwgIndvcmtwbGFjZV9wZWVyX3JldmlldyIpKSwKICBkb2N1bWVudGF0aW9uMC53aXRoKGMoIndvcmtfZG9tYWluIiwgIndvcmtwbGFjZV90ZF90cmFja2luZyIpKSwKICAKICBkb2N1bWVudGF0aW9uMC53aXRoKGMoIndvcmtwbGFjZV9wZWVyX3JldmlldyIsICJ3b3JrcGxhY2VfdGRfdHJhY2tpbmciKSksCiAgCiAgZG9jdW1lbnRhdGlvbjAud2l0aChjKCJ3b3JrX2RvbWFpbiIsICJ3b3JrcGxhY2VfcGVlcl9yZXZpZXciLCAid29ya3BsYWNlX3RkX3RyYWNraW5nIikpCikKYGBgCiMjIENhbmRpZGF0ZSBtb2RlbHMKV2UgaW5zcGVjdCBzb21lIG9mIG91ciB0b3AgcGVyZm9ybWluZyBtb2RlbHMuIAoKQWxsIG1vZGVscyBzZWVtcyB0byBoYXZlIHNhbXBsZWQgbmljZWx5IChyaGF0IGlzIGNhIDEgYW5kIGZsdWZmeSBwbG90cykgdGhleSBhbHNvIGhhdmUgYWJvdXQgdGhlIHNhbWUgZml0IHRvIHRoZSBkYXRhIGVuZCBzaW1pbGFyIGVzdGltYXRlZCBmb3IgdGhlIGhpZ2hfZGVidF92ZXJzaW9uIGJldGEgcGFyYW1ldGVyCgoKYGBge3J9CmRvY3VtZW50YXRpb24wIDwtIGRvY3VtZW50YXRpb24wLndpdGgoKQpzdW1tYXJ5KGRvY3VtZW50YXRpb24wKQpyYW5lZihkb2N1bWVudGF0aW9uMCkKcGxvdChkb2N1bWVudGF0aW9uMCwgYXNrID0gRkFMU0UpCgpwcF9jaGVjayhkb2N1bWVudGF0aW9uMCwgbnNhbXBsZXMgPSAxNTAsIHR5cGU9ICJiYXJzIikKCmBgYApgYGB7cn0KZG9jdW1lbnRhdGlvbjEgPC0gZG9jdW1lbnRhdGlvbjAud2l0aCgid29ya3BsYWNlX3BlZXJfcmV2aWV3IikKc3VtbWFyeShkb2N1bWVudGF0aW9uMSkKcmFuZWYoZG9jdW1lbnRhdGlvbjEpCnBsb3QoZG9jdW1lbnRhdGlvbjEsIGFzayA9IEZBTFNFKQoKcHBfY2hlY2soZG9jdW1lbnRhdGlvbjEsIG5zYW1wbGVzID0gMTUwLCB0eXBlPSAiYmFycyIpCgpgYGAKCmBgYHtyfQpkb2N1bWVudGF0aW9uMiA8LSBkb2N1bWVudGF0aW9uMC53aXRoKGMoIndvcmtwbGFjZV9wZWVyX3JldmlldyIsICJ3b3JrX2RvbWFpbiIpKQpzdW1tYXJ5KGRvY3VtZW50YXRpb24yKQpyYW5lZihkb2N1bWVudGF0aW9uMikKcGxvdChkb2N1bWVudGF0aW9uMiwgYXNrID0gRkFMU0UpCgpwcF9jaGVjayhkb2N1bWVudGF0aW9uMiwgbnNhbXBsZXMgPSAxNTAsIHR5cGU9ICJiYXJzIikKCmBgYAoKYGBge3J9CmRvY3VtZW50YXRpb24zIDwtIGRvY3VtZW50YXRpb24wLndpdGgoYygid29ya3BsYWNlX3BlZXJfcmV2aWV3IiwgIndvcmtfZG9tYWluIiwgIndvcmtwbGFjZV90ZF90cmFja2luZyIpKQpzdW1tYXJ5KGRvY3VtZW50YXRpb24zKQpyYW5lZihkb2N1bWVudGF0aW9uMykKcGxvdChkb2N1bWVudGF0aW9uMywgYXNrID0gRkFMU0UpCgpwcF9jaGVjayhkb2N1bWVudGF0aW9uMywgbnNhbXBsZXMgPSAxNTAsIHR5cGU9ICJiYXJzIikKCmBgYAoKIyMjIEZpbmFsIG1vZGVsCkFsbCBtb2RlbHMgaGF2ZSBzYW1wbGVkIG9rZXkgYnV0IGRvY3VtZW50YXRpb24zIGhhcyBhIGJpdHRlciBmaXQgZm9yIHRoZSBoaWdoIGRlYnQgYmV0YSBwYXJhbWV0ZXIgaXMgc2lnbmlmaWNhbnRseSBiZXR0ZXIgdGhhbiBzb21lIG9mIHRoZSBzaW1wbGUgbW9kZWxzIGFjY29yZGluZyB0byBsb28uIHdlZSBjaG9vc2UgZG9jdW1lbnRhdGlvbjMgYXMgb3V0IGZpbmFsIG1vZGVsLgoKIyMjIFRyeWluZyB0byBhZGQgY292YXJpYW5jZQoKYGBge3J9CmRvY3VtZW50YXRpb24zX3dpdGhfYyA8LSBicm0oCiAgImRvY3VtZW50YXRpb24gfiAxICsgaGlnaF9kZWJ0X3ZlcnNpb24gKyB3b3JrcGxhY2VfcGVlcl9yZXZpZXcgKyB3b3JrX2RvbWFpbiArIHdvcmtwbGFjZV90ZF90cmFja2luZyArICgxIHwgYyB8IHNlc3Npb24pIiwKICBwcmlvciA9IGMoCiAgICBwcmlvcihub3JtYWwoMCwgMC45KSwgY2xhc3MgPSAiYiIpLAogICAgcHJpb3Iobm9ybWFsKDAsIDAuOSksIGNsYXNzID0gIkludGVyY2VwdCIpLAogICAgcHJpb3IoZXhwb25lbnRpYWwoMSksIGNsYXNzID0gInNkIiwgZHBhciA9ICJtdUluY29ycmVjdCIpLAogICAgcHJpb3IoZXhwb25lbnRpYWwoMSksIGNsYXNzID0gInNkIiwgZHBhciA9ICJtdU5vbmUiKSwKICAgIHByaW9yKGxraigyKSwgY2xhc3MgPSAiTCIpCiAgKSwKICBmYW1pbHkgPSBjYXRlZ29yaWNhbCgpLAogIGRhdGEgPSBhcy5kYXRhLmZyYW1lKGQuYm90aF9jb21wbGV0ZWQpLAogIGZpbGUgPSAiZml0cy9kb2N1bWVudGF0aW9uM193aXRoX2MiLAogIGZpbGVfcmVmaXQgPSAib25fY2hhbmdlIgopCgpzdW1tYXJ5KGRvY3VtZW50YXRpb24zX3dpdGhfYykKcmFuZWYoZG9jdW1lbnRhdGlvbjNfd2l0aF9jKQpwbG90KGRvY3VtZW50YXRpb24zX3dpdGhfYywgYXNrID0gRkFMU0UpCgpwcF9jaGVjayhkb2N1bWVudGF0aW9uM193aXRoX2MsIG5zYW1wbGVzID0gMTUwLCB0eXBlPSAiYmFycyIpCmBgYAoKYGBge3J9CmxvbygKICBkb2N1bWVudGF0aW9uMywKICBkb2N1bWVudGF0aW9uM193aXRoX2MKKQpgYGAKQWRkaW5nIHRoZSBjb3ZhcmlhbmNlIHBhcmFtZXRlciBkaWQgbm90IHNpZ25pZmljYW50bHkgaW1wcm92ZSB0aGUgbW9kZWwgYW5kIHdlIHdpbGwgdGhlcmVmb3JlIGNvbnRpbnVlIHRvIHVzZSB0aGUgc2ltcGxlciBtb2RlbCBkb2N1bWVudGF0aW9uMy4KCiMjIyBNb2RlbCB3aXRoIHdpdGggaW5jb21wbGV0ZSBkYXRhIHBvaW50cwoKU29tZSBwYXJ0aWNpcGFudHMgZGlkIG9ubHkgY29tcGxldGUgb25lIHNjZW5hcmlvLiBUaG9zZSBoYXMgYmVlbiBleGNsdWRlZCBmcm9tIHRoZSBpbml0aWFsIGRhdGFzZXQgdG8gaW1wcm92ZSBzYW1wbGluZyBvZiB0aGUgbW9kZWxzLiBXZSBkbyBob3dldmVyIHdhbnQgdG8gdXNlIGFsbCBkYXRhIHdlIGNhbiBhbmQgd2lsbCB0aGVyZWZvcmUgdHJ5IHRvIGZpdCB0aGUgbW9kZWwgd2l0aCB0aGUgY29tcGxldGUgZGF0YXNldC4KCmBgYHtyfQpkb2N1bWVudGF0aW9uMy5hbGwgPC0gYnJtKAogICJkb2N1bWVudGF0aW9uIH4gMSArIGhpZ2hfZGVidF92ZXJzaW9uICsgd29ya3BsYWNlX3BlZXJfcmV2aWV3ICsgd29ya19kb21haW4gKyB3b3JrcGxhY2VfdGRfdHJhY2tpbmcgKyAoMSB8IHNlc3Npb24pIiwKICBwcmlvciA9IGMoCiAgICBwcmlvcihub3JtYWwoMCwgMC45KSwgY2xhc3MgPSAiYiIpLAogICAgcHJpb3Iobm9ybWFsKDAsIDAuOSksIGNsYXNzID0gIkludGVyY2VwdCIpLAogICAgcHJpb3IoZXhwb25lbnRpYWwoMSksIGNsYXNzID0gInNkIiwgZHBhciA9ICJtdUluY29ycmVjdCIpLAogICAgcHJpb3IoZXhwb25lbnRpYWwoMSksIGNsYXNzID0gInNkIiwgZHBhciA9ICJtdU5vbmUiKQogICksCiAgZmFtaWx5ID0gY2F0ZWdvcmljYWwoKSwKICBkYXRhID0gYXMuZGF0YS5mcmFtZShkLmNvbXBsZXRlZCksCiAgZmlsZSA9ICJmaXRzL2RvY3VtZW50YXRpb24zLmFsbCIsCiAgZmlsZV9yZWZpdCA9ICJvbl9jaGFuZ2UiCikKc3VtbWFyeShkb2N1bWVudGF0aW9uMy5hbGwpCnJhbmVmKGRvY3VtZW50YXRpb24zLmFsbCkKcGxvdChkb2N1bWVudGF0aW9uMy5hbGwsIGFzayA9IEZBTFNFKQoKcHBfY2hlY2soZG9jdW1lbnRhdGlvbjMuYWxsLCBuc2FtcGxlcyA9IDE1MCwgdHlwZT0gImJhcnMiKQpgYGAKClRyYWluaW5nIHRoZSBtb2RlbCBvbiBhbGwgZGF0YSBwb2ludHMgcmVkdWNlcyB0aGUgdW5jZXJ0YWludHkgYW5kIGRpZCBub3QgcmVzdWx0IGluIGFueSBzYW1wbGluZyBwcm9ibGVtcy4gV2Ugd2lsbCBwcm9jZWVkIHdpdGggdGhlIG1vZGVsIGZpdHRlZCB0byBhbGwgdGhlIGRhdGEuCgojIyBJbnRlcnByZXRpbmcgdGhlIG1vZGVsCgpFeHRyYWN0IHBvc3RlcmlvciBzYW1wbGVzOgpgYGB7cn0KCnBvc3QgPC0gcG9zdGVyaW9yX3ByZWRpY3QoZG9jdW1lbnRhdGlvbjMuYWxsLCBuZXdkYXRhID0gZGF0YS5mcmFtZSgKICBoaWdoX2RlYnRfdmVyc2lvbiA9IGMoImZhbHNlIiwgInRydWUiKSwKICBzZXNzaW9uID0gTkEsIAogIHdvcmtwbGFjZV9wZWVyX3JldmlldyA9IE5BLCAKICB3b3JrX2RvbWFpbiA9IE5BLCAKICB3b3JrcGxhY2VfdGRfdHJhY2tpbmcgPSBOQQopKQpwb3N0LmxvdyA8LSAgcG9zdFssMV0KcG9zdC5oaWdoIDwtIHBvc3RbLDJdCnN1bW1hcnkocG9zdCkKCmBgYAo=